Index-based sampling policies for tracking dynamic networks under sampling constraints

Abstract

We consider the problem of tracking the topology of a large-scale dynamic network with limited monitoring resources. By modeling the dynamics of links as independent ON-OFF Markov chains, we formulate the problem as that of maximizing the overall accuracy of tracking link states when only a limited number of network elements can be monitored at each time step. We consider two forms of sampling policies: link sampling, where we directly observe the selected links, and node sampling, where we observe states of the links adjacent to the selected nodes. We reduce the link sampling problem to a Restless Multi-armed Bandit (RMB) and prove its indexability under certain conditions. By applying the Whittle's index policy, we develop an efficient link sampling policy with methods to compute the Whittle's index explicitly. Under node sampling, we use a linear programming (LP) formulation to derive an extended policy that can be reduced to determining the nodes with maximum coverage of the Whittle's indices. We also derive performance upper bounds in both sampling scenarios. Simulations show the efficacy of the proposed policies. Compared with the myopic policy, our solution achieves significantly better tracking performance for heterogeneous links.

Full text

Page 1

Index-Based Sampling Policies for Tracking
Dynamic Networks under Sampling Constraints
Ting He∗, Animashree Anandkumar†, and Dakshi Agrawal∗.
∗IBM T.J. Watson Research Center, Hawthorne, NY, USA. Email: {the,agrawal}@us.ibm.com
†Dept. of EECS, University of California Irvine, Irvine, CA, USA. Email: {a.anandkumar}@uci.edu
Abstract—We consider the problem of tracking the topology of
a large-scale dynamic network with limited monitoring resources.
By modeling the dynamics of links as independent ON-OFF
Markov chains, we formulate the problem as that of maximizing
the overall accuracy of tracking link states when only a limited
number of network elements can be monitored at each time step.
We consider two forms of sampling policies: link sampling, where
we directly observe the selected links, and node sampling, where
we observe states of the links adjacent to the selected nodes.
We reduce the link sampling problem to a Restless Multi-armed
Bandit (RMB) and prove its indexability under certain conditions.
By applying the Whittle’s index policy, we develop an efficient
link sampling policy with methods to compute the Whittle’s index
explicitly. Under node sampling, we use a linear programming
(LP) formulation to derive an extended policy that can be reduced
to determining the nodes with maximum coverage of the Whittle’s
indices. We also derive performance upper bounds in both
sampling scenarios. Simulations show the efficacy of the proposed
policies. Compared with the myopic policy, our solution achieves
significantly better tracking performance for heterogeneous links.
Index Terms—Network sampling, network topology tracking,
restless multi-armed bandits, Whittle’s index policy.
I. INTRODUCTION
Large-scale dynamic networks are widely used to model
relationships and interactions among entities in domains as di-
verse as IT systems, sociology, and economics. A fundamental
challenge in managing large dynamic networks is designing
sampling policies so that a network can be efficiently moni-
tored. A concrete example comes from the cloud computing
paradigm where large data centers nimbly shuffle load between
tens of thousands of servers to optimize server utilization.
The monitoring infrastructure in such data centers may gather
hundreds of metrics from each server which are then sent
to a central location for further analysis. Such analysis often
involves determining whether certain relationship (e.g., time-
lagged causality of crossing certain thresholds) exists between
the metrics. The entire set of relationships can be modeled
as a logical network of the metrics where links denote the
presence of the pairwise relationships. Due to the large number
of metrics involved, it may not be feasible to assess all the
relationships at all times, and we need to reduce the problem
size by sampling the network. This raises several questions:
How do we estimate the unobserved link states when tracking
The first and the third authors are supported in part by the U.S. National Institute
of Standards and Technology under Agreement Number 60NANB10D003. The second
author is supported in part by a MURI funded through ARO Grant W911NF-06-1-0076.
a sampled dynamic network? Given sampling constraints, how
do we sample the network to maximize tracking accuracy?
Although our research was initially motivated by this scenario,
these questions are fundamental in many other application
domains as well.
Decisions on which network nodes or links (henceforth
referred to as network elements) should be selected for
monitoring critically depend on the nature of their temporal
evolution. Intuitively, slowly-evolving elements can
sampled at a lower rate, and hence the monitoring resources
originally dedicated to them can be deployed elsewhere
to better keep track of rapidly-changing elements. In this
work, we address the following issues: Are there provably
efficient sampling policies that explicitly take into account
the monitoring constraints and the temporal evolution of the
network? Which networks are amenable for tracking and how
does the network topology affect the tracking accuracy? The
answers to these questions will provide important insights
towards engineering real-world solutions.
be
A. Our Approach & Contributions
We model the network as a graph G = (V,E) in which
a vertex v ∈ V represents an entity of interest and a link
e ∈ E represents a pairwise interaction between two entities
that we would like to monitor. In our formulation, the set
of vertices V is fixed and each link in E has a binary state
(“ON” or “OFF”) which evolves according to a discrete-time
Markov chain, independent of other links. We consider two
scenarios, viz., link sampling and node sampling. Under link
sampling, we assume that we can sample at most K links
at a time to determine their states. Under node sampling, we
assume that we can sample at most K nodes at a time to
determine the states of all their adjacent links. In both cases,
the states of unobserved links need to be estimated from the
past observations. Our goal is to track the states of all the
links in E with maximum accuracy under given sampling
constraints. In this setting, our contributions are as follows:
• Theoretical framework: We
framework for optimally tracking the link states of dynamic
networks through either link or node sampling. We apply
Bayesian filtering and maximum a posteriori (MAP) estimation
to optimally estimate the states of unsampled links and design
sampling policies to maximize the estimation accuracy under
sampling constraints.
develop a systematic

Page 2

• Efficient solutions: Under link sampling, we reduce the
problem to the well-known Restless Multi-armed Bandit
(RMB) [1]. The RMB is known to be PSPACE-hard in
general [2] unless it is indexable, in which case the Whittle’s
index policy [1] is known to be an efficient solution. We
establish indexability of our problem and develop an efficient
algorithm to compute the Whittle’s index. Under node
sampling, we show that our problem cannot be reduced
to RMB. Instead, we formulate a linear program (LP)
under relaxed sampling constraints, a technique inspired by
Whittle’s original formulation [1]. This yields an extended
Whittle’s index policy which selects nodes that maximize the
sum of the Whittle’s indices at covered links. A byproduct of
our computation is a procedure to compute upper bounds on
the optimal performance in both sampling scenarios.
• Numerical studies: Our simulations demonstrate the effi-
cacy of the proposed solutions. We show that the Whittle’s in-
dex policy is near-optimal for link sampling, but has a gap (up
to 10%) when compared with the upper bound for node sam-
pling. Compared with the myopic policy, the (extended) Whit-
tle’s index policy performs similarly for i.i.d. links but increas-
ingly better (up to more than 20%) as link activities diverge.
We note a limiting assumption in our model—that each
link evolves independently according to a Markov chain. This
may not be true in certain scenarios, e.g., in a social network,
transitivity of relationships is likely to cause correlated
link evolution, in which case this assumption precludes the
application of our techniques. Nonetheless, our model captures
a fundamental aspect of network evolution and provides a
sound foundation for further development of network
sampling techniques in more general scenarios. To the best of
our knowledge, this is the first systematic study of tracking
dynamic networks under monitoring constraints that explicitly
takes into account the temporal evolution of the networks.
B. Related Work
Learning static network topology has received considerable
attention, with extensive studies including those on sampling
social networks [3], mapping the Internet [4], and network
tomography in a variety of settings. Diverse measurements
have been used to infer network structures, such as passive
measurements [5], node queries [6], edge counts [7], and
subgraph counts [8]. However, these works all assume static
networktopologies,andlearning
networks is a relatively unexplored area. A related problem
of tracking dynamic bipartite graphs is considered in [9] but
only to track the proximity measures between nodes rather
than the graph itself, without any sampling constraints.
Our solution is based on Restless Multi-armed Bandits
(RMBs), which have been extensively studied since the sem-
inal work of Whittle [1], where an index-based policy is pro-
posed for a special class of RMBs known as indexable RMBs.
It was later proved that finding an optimal policy for a general
RMB, where indexability may not hold, is PSPACE-hard [2].
In [10], sufficient conditions for indexability are derived. In
[11], conditions for asymptotic optimality of the Whittle’s
dynamically-changing
active link
inactive link
0
1
e
pe
00
pe
01
pe
10
pe
11
base network
Fig. 1.
a Markov chain; the true topology at any time only contains the active links.
A dynamic network: each dynamic link changes states over time by
index policy are established. A special class of RMBs based
on ON-OFF Markov chains has been extensively studied in the
context of UAV routing [12] and dynamic channel access [13],
where the goal is to access arms when they are in “ON” states.
Our work uses a similar model, but is fundamentally different
in that we want to track the states of all the arms, leading to
very different solutions and conclusions as shown later.
The rest of the paper is organized as follows. Section II for-
mulates the problem, Section III presents some preliminaries,
Section IV and Section V deal with link and node sampling,
respectively, Section VI gives simulation results, and Section
VII concludes the paper.
II. PROBLEM FORMULATION
Notations: For set A, let |A| denote its cardinality. We use
the convention that X denote the random variable, x the real-
ization, and X (or x) the vector versions. Let I denote the in-
dicator function. We will use “arm” and “link” interchangably.
A. Dynamic Networks and Temporal Process
As illustrated in Fig. 1, let G = (V,E) denote a base
network over a node set V , in which the links in E are active
intermittently. Let N := |V | and M := |E| be the numbers
of nodes and links. Given a node v ∈ V , we use N(v;G)
(N(v) for short) to denote the set of its adjacent links in G.
The network evolves in discrete time1and let Gt= (Vt,Et)
be the network at time t. We assume that the set of nodes is
fixed (Vt≡ V ), but the set of active links changes dynamically.
Using Se(t) ∈ {0,1} to denote the state of link e ∈ E at time
t, we have Se(t) = 1 (“active”) for e ∈ Et and Se(t) = 0
(“inactive”) otherwise. To model the temporal dynamics of the
network, we assume that the link state Se(t) evolves according
to a discrete-time Markov chain, independent of the other
links. Denote the transition probabilities by
pe
pe
01:= P[Se(t) = 1|Se(t − 1) = 0],
11:= P[Se(t) = 1|Se(t − 1) = 1].
(1)
(2)
Assume pe
01and pe
11are known for all the links e ∈ E.
1The problem can also be formulated in continuous time.

Page 3

B. Sampling Constraints
For large networks, it may not be feasible to check the
states of all links simultaneously, e.g., this requires correlating
the sequences of measurements for each pair of metrics in
a cloud dependency network. This imposes constraints on
concurrent sampling. We consider two sampling scenarios,
viz., link sampling and node sampling. Under link sampling,
we assume that at most K links can be sampled at any time
and their states can be directly observed. Under node sampling,
at most K nodes can be sampled at a time and the states of
all the links adjacent to these nodes can be directly observed.
In both cases, the states of the unobserved links need to be
predicted from past observations.
Let π = (πt)t∈Ndenote a policy which samples the network
under the given constraints. Let Eπt⊂ E denote the subset
of links which are directly observed. Under link sampling, it
is required that |Eπt| ≤ K for all t ∈ N, while under node
sampling, it is required that Eπtcan be covered by at most K
nodes. Let ΠE,Kand ΠV,Kbe the set of valid link and node
sampling policies, respectively.
Let Zπ
e(t) be the observed state of link e at time t, i.e.,
Zπ
Zπ
e(t) := (Zπ
time t. For the links that are not observed at a given time, we
can predict their current state based on the past observations.
Let?Se(t) : Zπ
C. Optimal Tracking of Dynamic Networks
e(t) = Se(t) if e ∈ Eπtand Zπ
e(s))t
e(t) = ∅ otherwise. Let
s=1be the observation history of link e until
e(t) ?→ {0,1} be the estimated state.
Our goal is to track the link states with the maximum
accuracy under given sampling constraints. To this end, we
measure the reward of a sampling policy π at time t by the
expected number of correctly tracked links:
?
e∈E
R(t;π) :=
P[Se(t) =?Se(t)].
(3)
For the overall reward, we consider the discounted case with
discount factor β, where the long-term reward is given by2
R∞(π) := Eπ[
∞
?
t=1
βt−1R(t;π)],β ∈ [0,1).
(4)
We want to find policies which maximize the tracking accuracy
measured by (4) under link or node sampling constraints:
π∗
E,K:= argmax
π∈ΠE,K
R∞(π),π∗
V,K:= argmax
π∈ΠV,K
R∞(π), (5)
III. PRELIMINARIES
A. MAP Estimation
Under a fixed sampling policy π, we now derive the optimal
estimator for unobserved links. For any link e ∈ E, given
the past and current observations Zπ
e(t), the optimal estimator
2Strictly, (4) should also be a function of the initial state x(1), but we
suppress this in the sequel for convenience.
which minimizes the error probability is the maximum a
posteriori (MAP) estimator [14]
?
0
o.w.
It is clear that when a link is observed at time t, i.e., e ∈ Eπt,
then?Se(t) = Se(t) and there is no tracking error.
mator can be efficiently implemented using a Bayesian filter.
Let Xπ
e(t − 1)] denote the belief that
the link is in the active state given all the past observations.
The update rule for the belief is given by


where the mapping Te(x) := (pe
passive update under the Markovian rule. Hence, the tracking
accuracy under MAP estimation is given by
?1,
max(Xπ
?Se(t) =
1,
if P[Se(t) = 1|Zπ
e(t)] ≥ 0.5, (6)
Under the Markovian link model in (1–2), the MAP esti-
e(t) := P[Se(t) = 1|Zπ
Xπ
e(t) =

pe
pe
Te(Xπ
11
if Zπ
if Zπ
if Zπ
e(t − 1) = 1,
e(t − 1) = 0,
e(t − 1) = ∅,
01
e(t − 1))
(7)
11− pe
01)x + pe
01denotes a
P[?Se(t) = Se(t)] =
It thus suffices to evaluate R(t;π) in (3) by (8).
if e ∈ Eπt,
e(t),1 − Xπ
e(t)), o.w.(8)
B. Recap of Restless Multiarmed Bandits
Since our problem is closely related to the Restless Multi-
armed Bandits (RMB), we first provide a brief recap. In a
RMB [1], M stochastic arms are given, where each arm e is an
independently evolving Markov chain with different transition
matrices PU
e(i,j)] under active (U = 1) and passive
(U = 0) actions. Given rewards RU
when e is in state x, the goal is to select actions such that the
total reward as in (4) is maximized, under the constraint that
only K arms can be activated at each time.
Whittle [1] proposed an efficient index-based policy for a
limited class of RMBs which are called indexable. The policy
assigns an index to each arm at every time step depending
on its current state, and activates the arms with the K largest
indices. It is a simple policy with only O(M) computational
complexity at each step. To define indexability and Whittle’s
index, we introduce a notion of subsidy for passivity: under
subsidy m, an arm earns an extra reward m for being passive,
i.e., the passive reward becomes m+R0
set P(m) as the set of states x where the passive action is
optimal under subsidy m.
Definition 1 (Indexability of RMB): A stochastic arm is in-
dexable if the passive set P(m) monotonically increases from
the empty set to the full state space as m increases from −∞
to ∞. An RMB is indexable if every arm is indexable.
Definition 2 (Whittle’s Index): For an indexable arm in
state x, the Whittle’s index W(x) is defined as
e= [PU
e(x) for taking action U
e(x). Define a passive
W(x) := inf
m∈R{m : x ∈ P(m)}.
(9)
Hence, the Whittle’s index for a state is the minimum
subsidy needed to make it worthwhile to be passive (assuming
optimal actions in the future). Under regularity conditions, the

Page 4

Whittle’s index policy is asymptotically optimal for i.i.d. arms
as M → ∞ and K/M → c (a constant) [11]. Empirical
studies have also demonstrated its superior performance in
various applications (see [13]). The Whittle’s index policy is
thus considered a simple and effective solution for indexable
RMBs. On the other hand, a general RMB is PSPACE-hard
[2]. Hence, establishing indexability and finding the Whittle’s
index are crucial for solving RMBs efficiently.
IV. NETWORK TRACKING UNDER LINK SAMPLING
We now cast the link sampling problem as a RMB. The
node sampling problem will be considered later in Section V.
A. Restless Bandit Formulation
Each arm corresponds to a link e ∈ E, and the belief
Xe(t) ∈ [0,1] defined as in (7) (π has been dropped)
denotes the state of arm e at time t. An active action is
taken (Ue(t) = 1) if e is sampled; otherwise, it is passive
(Ue(t) = 0). We can thus rewrite the update of Xe(t) based
on the sampling action3


The transition probability from belief y to belief x is thus
?Ix=Te(y),
yIx=pe
Xe(t + 1) =

pe
pe
11
w.p. Xe(t)
w.p. 1 − Xe(t)
01
?
if Ue(t) = 1,
Te(Xe(t))
if Ue(t) = 0.
(10)
PU
e(y,x) =
if U = 0,
11+ (1 − y)Ix=pe
01,
o.w.,(11)
and the one-step reward is given by
R1
e(x) = 1, R0
e(x) = max(x,1 − x).
(12)
Under subsidy m, the passive reward will be replaced by
R0
e(x) = m + max(x, 1 − x).
A subtle difference between our problem and a classic RMB
is that our reward in (12) does not depend on the actual link
state Se(t), but on the belief Xe(t). Thus, our problem is
actually an RMB on uncountable state space even though the
underlying link state is binary.
B. Myopic Sampling Policy
Before we proceed to computing the Whittle’s index policy
for the above RMB, we first present a simple index policy
called the myopic policy. It is a greedy policy which maximizes
the immediate reward in the current time step. For the RMB
for link sampling, it assigns an index
Ye(x) := min(x,1 − x)
(13)
for each link in state x and picks the K links with the largest
indices. This policy is always applicable (i.e., the RMB need
not be indexable) and easy to compute. The drawback is
that it is suboptimal in general. We have the following result
comparing it to the Whittle’s index.
Proposition 1: For the link sampling problem, the Whittle’s
index (if exists) is no smaller than the myopic index, i.e.,
3“w.p.” means “with probability”.
W(x) ≥ Y (x) ∀x ∈ [0, 1]. Moreover, W(x) → Y (x) as
|p11− p01| → 0.
Proof:
See [15].
For the special case of pe
becomes i.i.d. over time, and it can be shown that the Whittle’s
index policy and the myopic policy are equivalent and both
optimal. The two will be further compared in Section VI.
2
11= pe
01, the link state Se(t)
C. Indexability of the RMB
Since the Whittle’s index policy is only valid for indexable
arms, it is crucial to establish indexability of the above RMB.
We will focus on one arm and drop the subscript e. Define
the value function Vβ,m(x) as the maximum total discounted
reward for an arm initially at state x under discount β and
subsidy m. By definition, it satisfies
Vβ,m(x) = max(Vβ,m(x;U = 0),Vβ,m(x;U = 1)),
(14)
where Vβ,m(x;U) is the optimal discounted reward if taking
action U in the first time step. By substituting for the reward
functions in (12), we have
Vβ,m(x;U=0) = m + max(x, 1 − x) + βVβ,m(T (x)),
Vβ,m(x;U=1) = 1 + β (xVβ,m(p11) + (1 − x)Vβ,m(p01)).
(15)
(16)
Equations (14–16) jointly give an equation of the (unknown)
value function Vβ,m(x), called the Bellman equation. Theo-
retically, once we solve the Bellman equation for Vβ,m(x),
we can obtain the optimal policy for the single-armed bandit
under subsidy m and the associated passive set as
π∗
m(x) = argmax
U=0,1
Vβ,m(x; U),
(17)
P(m) = {x : Vβ,m(x; U = 0) ≥ Vβ,m(x; U = 1)}.
(18)
We now test the indexability of the link sampling problem
by comparing the passive set (18) with the conditions in
Definition 1. We first show the following property of P(m).
Lemma 1: The optimal policy for the single-armed bandit
with subsidy m is a threshold policy: π∗
τ−(m) < x < τ+(m), i.e., P(m) = [0, τ−(m)]∪[τ+(m), 1].
Proof: See [15].
V
m(x) = 1 if and only if
x
01
τ−(m)
τ+(m)
Vβ,m(x; U = 1)
(a)
(b)
(c)
P(m)
Fig. 2.
Vβ,m(x; U = 0) for (a) m > 0, (b) m = 0, and (c) m < 0.
The thresholds τ−(m), τ+(m) are intersecting points be-
tween Vβ,m(x; U = 0) and Vβ,m(x; U = 1) (see [15]), as
illustrated in Fig. 2. This structure leads us to analyzing the
properties of τ−(m) and τ+(m), with the following result.
Threshold structure of the optimal policy: value function

Page 5

Lemma 2: The thresholds τ−(m), τ+(m) are monotone
increasing and decreasing, respectively, with m for β ≤ 0.5.
Proof: See [15].
We now prove the indexability of our problem.
Theorem 1: The RMB for link sampling is indexable if the
discount factor β ≤ 0.5.
Proof: By the proof of Lemma 1, we see that P(m) = ∅
for m < 0 and P(m) = [0,1] for m > mmax. For m ∈ [0,mmax],
Lemma 2 implies that P(m) is monotone increasing with m.
Hence, Definition 1 is satisfied.
In fact, numerical calculation (Fig. 4) shows that the
monotonicity of τ−(m), τ+(m) holds for β > 0.5 as well.
We conjecture that the RMB is indexable for all β ∈ [0,1).
D. Computation of Whittle’s Index
By definition (9), the Whittle’s index of state x is the
subsidy m such that τ−(m) = x or τ+(m) = x. To eval-
uate τ−(m), τ+(m), it is necessary to compute Vβ,m(x;U)
and thus the value function. The problem is that there are
uncountably many beliefs, making it impossible to directly
solve the Bellman equation. In this section, we will provide
an alternative procedure to compute the Whittle’s index.
1) Preliminaries: We first prepare some tools. Define
Tl(x):=Pr{S(1 + l) = 1|X(1) = x} as the l-step transition
of a passive arm. It is easy to show that
Tl(x) = x0− (x0− x)(p11− p01)l,
(19)
where x0:=p01/(1 + p01− p11) is the equilibrium distribution
of the arm. As shown in [13], when l → ∞, Tl(x) con-
verges to x0monotonically if p11> p01(called a positively-
correlated arm), or alternatively from above and below if
p11< p01(called a negatively-correlated arm). Now define
L(x; c1, c2):=min{l : Tl(x) ∈ (c1, c2)}
(20)
to be the hitting time of the interval (c1, c2) from initial state
x under transition Tl. We can calculate the hitting time in
closed form as in [15].
2) Computing Value Function: Next, we derive a method
to compute the value function directly. Due to the threshold
structure of the optimal single-armed policy (Lemma 1), the
belief evolution has the following pattern: starting from initial
state x, it passively transits L(x; τ−(m), τ+(m)) =: L steps
to T (x), T2(x),... until hitting the active set TL(x) ∈
(τ−(m), τ+(m)), at which time the arm will be activated
(sampled) and the state reset to p01 or p11. Accordingly, we
can decompose the total value into two parts: (i) passive re-
wards and subsidies for steps 0,..., L−1, and (ii) total future
rewards from step L onward. Part (i) can be easily computed as
(1−βL)m
1−β
Ti(x)) can be computed in closed form as in [15]. Part (ii) is
simply βLVβ,m(TL(x); U = 1). Therefore, given m, τ−(m),
and τ+(m), we can compute the value function by
+f(x; L), where f(x; L):=
L−1
?
i=0
βimax(Ti(x), 1−
Vβ,m(x) =(1 − βL)m
1 − β
+ f(x; L) + βL
+βL+1TL(x)Vβ,m(p11) + βL+1(1 − TL(x))Vβ,m(p01).(21)
The only unknowns left are Vβ,m(p11), Vβ,m(p01). Note that
x = p11 or p01 should also satisfy (21), giving us two
equations with two unknowns. Solving these equations yields
Vβ,m(p11), Vβ,m(p01); see [15] (equations (29–30)).
3) Computing Whittle’s Index: We are now ready to com-
pute the Whittle’s index. First, we note that although seem-
ingly complicated, the expression of value function given by
(21) has a simple linear form in m for fixed thresholds.
Lemma 3: Given thresholds τ−(m) and τ+(m), for coef-
ficients ai, bi (i = 1, 2) as in [15] (equations (3–6)), define
functions
1 − βL(x)
1 − β
+βL(x)+1(1 − TL(x)(x))a1,
b(x) :=f(x; L(x)) + βL(x)+ βL(x)+1TL(x)(x)b2
+βL(x)+1(1 − TL(x)(x))b1,
a(x):=
+ βL(x)+1TL(x)(x)a2
(22)
(23)
where L(x):=L(x; τ−(m), τ+(m)). Then the value function
is equal to Vβ,m(x) = a(x)m + b(x), with end values
Vβ,m(p01) = a1m + b1, and Vβ,m(p11) = a2m + b2.
Proof: See [15].
Remarks: The above result reveals the following meanings
of a(x) and b(x): a(x) is the total discounted time for the arm
to be passive (under the optimal single-armed policy), as any
change in subsidy m results in an a(x)-fold change in the total
value; b(x) is the actual reward (excluding subsidies), i.e., the
discounted total number of times the arm is tracked correctly.
Based on Lemma 3, we can easily compute the subsidy m
for given τ−(m) and τ+(m). By definition, for x = τ−(m)
or τ+(m), Vβ,m(x; u = 0) = Vβ,m(x; u = 1), i.e.,
1 + βxVβ,m(p11) + β(1 − x)Vβ,m(p01)
= m + max(x, 1 − x) + βVβ,m(T (x)).
Substituting the linear forms of Vβ,m(·) into the above gives
m =1 + βxb2+ β(1 − x)b1− max(x,1 − x) − βb(T (x))
1 + βa(T (x)) − βxa2− β(1 − x)a1
=: M(x; τ−(m), τ+(m)),x = τ−(m), τ+(m). (24)
Given state x,
der subsidy m
then have W(x) = M(x; min(x, x′), max(x, x′)) =
M(x′; min(x, x′), max(x, x′)), where the only unknown
is x′. We can solve this equation (numerically) to compute x′
and hence W(x). The above gives a procedure to compute the
Whittle’s index as in lines (4–5) of Algorithm 1, which also
gives the corresponding index policy.
let x′
W(x) (x is also a threshold). We
be the other threshold un-
=
E. Upper Bound under Relaxed Link Sampling Constraint
A nice property of the Whittle’s index is that it is closely
related to a performance upper bound. Specifically, consider a
relaxed sampling constraint that only requires the discounted
average number of sampled links to be bounded by K. The
set of valid policies under the relaxed constraint is given by
?ΠE,K:= {π : Eπ[(1−β)
∞
?
t=1
βt−1?
e∈E
IUe(t)=1] ≤ K}. (25)