Coding perspectives for collaborative estimation over networks
ABSTRACT A collaborative distributed estimation problem over a communication constrained network is considered from an information theory perspective. A suitable architecture for the codes for this multiterminal information theory problem is determined under sourcechannel separation. In particular, distributed source codes in which each node multicasts a different message to each subset of other nodes are studied. This code construction hybridizes multiple description codes and codes for the CEO problem. The goal of this paper is to determine the fundamental relationship between the multicast communication rates and estimation performance obtainable. An achievable rate distortion region is proved to this problem and its structural properties are studied. Also, this achievable rate region is shown to simplify to the known bounds to some simpler problems.
 Citations (12)
 Cited In (0)

Article: A unified achievable rate region for a general class of multiterminal source coding systems
[show abstract] [hide abstract]
ABSTRACT: A unified treatment of a large class of multiterminal noiseless source coding problems including all previously studied situations is presented. A unified achievable rate region is established for this class by a coding technique based on the typical sequence criterion. This region is tight for all the previously studied situations.IEEE Transactions on Information Theory 06/1980; · 2.62 Impact Factor  SourceAvailable from: psu.edu[show abstract] [hide abstract]
ABSTRACT: We prove a new outer bound on the ratedistortion region for the multiterminal sourcecoding problem. This bound subsumes the best outer bound in the literature and improves upon it strictly in some cases. The improved bound enables us to obtain a new, conclusive result for the binary erasure version of the "CEO problem." The bound recovers many of the converse results that have been established for special cases of the problem, including the recent one for the Gaussian twoencoder problem.IEEE Transactions on Information Theory 06/2008; · 2.62 Impact Factor  SourceAvailable from: ifp.illinois.edu[show abstract] [hide abstract]
ABSTRACT: We consider a distributed sensor network in which several observations are communicated to the fusion center using limited transmission rate. The observation must be separately encoded so that the target can be estimated with minimum average distortion. We address the problem from an information theoretic perspective and establish the inner and outer bound of the admissible ratedistortion region. We derive an upper bound on the sumrate distortion function and its corresponding rate allocation schemes by exploiting the contrapolymatroid structure of the achievable rate region. The quadratic Gaussian case is analyzed in detail and the optimal rate allocation schemes in the achievable rate region are characterized. We show that our upper bound on the sumrate distortion function is tight for the quadratic Gaussian CEO problem in the case of same signaltonoise ratios at the sensors.IEEE Journal on Selected Areas in Communications 09/2004; · 3.12 Impact Factor
Page 1
Coding Perspectives for Collaborative Estimation Over
Networks
Sivagnanasundaram Ramanan and John MacLaren Walsh
Dept. of Electrical and Computer Engineering, Drexel University, Philadelphia, PA 19104, USA
Email: sur23@drexel.edu, jwalsh@coe.drexel.edu
Abstract—A collaborative distributed estimation problem over a com
munication constrained network is considered from an information theory
perspective. A suitable architecture for the codes for this multiterminal
information theory problem is determined under sourcechannel separa
tion. In particular, distributed source codes in which each node multicasts
a different message to each subset of other nodes are studied. This
code construction hybridizes multiple description codes and codes for the
CEO problem. The goal of this paper is to determine the fundamental
relationship between the multicast communication rates and estimation
performance obtainable. An achievable rate distortion region is proved
to this problem and its structural properties are studied. Also, this
achievable rate region is shown to simplify to the known bounds to
some simpler problems.
I. INTRODUCTION
Consider a network of M nodes deployed to monitor a common
phenomenon embodied by a sequence of random variables T(n).
Each node j ∈ [M] ({1,...,M} is denoted as [M]) in the network
makes indirect observations of this phenomenon, embodied as another
sequence of random variables Y(n)
j
the sequence
T(n),Y(n)
1
,...,Y(n)
M
probability distribution pT,Y1,...,YM.
Each node could use the local observations to obtain Bayesian
estimates ˜T(n)
j
of T(n)that minimize some local cost function
1
N
jj
j
is denoted as YN
communicate with each other in hopes of improving their estimates.
We will study such collaborative distributed estimation schemes
which accomplish this with separated network/channel and source
coding (despite the fact that such a separation is known to be
suboptimal in some multiterminal problems). The network/channel
codes see to it that messages sent over the network arrive at the
intended receivers unaltered, while the distributed source code sees
to it that the content of these messages provides the right information
extracted from the observations at a node in order to lower the
estimation error at the destination.
Our first insight, made in Section II, is that under this decompo
sition the proper source coding model reflecting the capabilities of
the network code is one in which each node multicasts a different
message to every possible subset of other nodes in the network. In
particular, the source encoder at each node j encodes its observations
YN
j
into a common message Qj→A ∈ {1,2,...,2NRj→A} to each
of the nodes with indices in some subset A of the other nodes
using an average of Rj→A bits per observation symbol. A different
such message can be encoded at each node j for each such subset
A ⊆ [M]\j, and then reliably multicasted (e.g. with the aid of some
S. Ramanan and J. M. Walsh were supported in part by the National Science
Foundation under grant CCF0728496 and part by the Air Force Office of
Scientific Research under grant FA955009C0014. They wish to thank Jun
Chen of McMaster University for helpful comments and suggestions in the
early stages of this work.
statistically related to T(n). Let
??
be i.i.d. according to joint
?N
n=1E
?
dj(˜T(n)
,T(n)
)??YN
?
(The vector
?
Y(1)
j
,...,Y(N)
j
?
j ). Alternatively, the nodes in the network could
Q1→2,3
Q1→2,3,Q2→1,3
Q2→1,3,Q3→1,2
Q3→1,2
Q2→1,3
Q1→2,3,Q3→1,2
Q2→1
Q1→2
Q2→3Q3→2
Q1→3Q3→1
1
23
Y1
Y2
Y3
Fig. 1.
3 nontrivial case of the problem of collaborative distributed estimation. To
determine the direction of travel of a message, note that the messages flow
in the direction in which they are read.
The “peace” network, which depicts the lowest dimensional M =
channel and network codes) to the nodes in A. For example, in the
lowest dimensional M = 3 nontrivial case, each node will create
multiple descriptions of its observations, one for each of the other
two nodes in the network individually, and one for both of them as
shown in Fig. 1.
We employ a classic technique from multiterminal information
theory [1] [2] to study the relationship between the rates {Rj→A j ∈
[M], A ∈ 2[M]\j} (2Sis the power set of subsets of S) of the source
code used, and the estimation errors Djthat each of the nodes can ob
tain in estimating the sequence T(n),n ∈ [N] from their own obser
vations YN
j
they have received.
One might view this model as a generalization of two classes
of multiterminal information theory problems: CEO problem [1],
[3], [4], [2], [5] and multiple descriptions problem [6]. The CEO
problem studies the rate  estimation error performance at the fusion
center, which estimates an underlying sequence of parameters solely
based on the rate constrained messages received from a collection
of nodes which independently encode using noisy local observations.
Two variations on the CEO problem are studied in [7] when decoder
side information [8] is available at the CEO and in [9] under the
name of manyhelpone problem when one of the nodes directly
observe the underlying sequence. In multiple descriptions problem
rate estimation error performance is studied in the case in which a
node encodes many descriptions of a source and sends different subset
of descriptions to different decoders which use those descriptions to
reproduce the source. A reader who is familiar with these two classes
of problems may question the purpose of studying this model when
both classes of problems are yet unsolved. Interestingly, we show
that some known bounds for the CEO problem and the multiple
and the messages QDj:=?Qi→A
??j ∈ A, A ∈ 2[M]\i?
Page 2
1
Y1
23
Y2 Y3
p(Y1,Y2,Y3 T)
Y1
Y2
Y3
T
p(T)
Fig. 2.
which only encodes a dedicated message to node 2 and a dedicated message
to node 3 is not general enough. Instead, the source encoder at node 1 should
encode a separate message for each possible subset of other nodes in the
network.
This network demonstrates that considering a source code at node 1
descriptions problem can be recovered from the results we derive
in this paper by hybridizing the techniques from both classes of
problems.
The paper is organized as follows. We discuss the best model for
the distributed source code in Section II. In Section III, we present
our main results on achievable rate distortion region. In Section IV,
we simplify the inner bound for some simpler problems.
II. DISTRIBUTED ESTIMATION AND MULTITERMINAL SOURCE
CODING
As outlined in the introduction, suppose we aim to separate the
source coding part of the distributed estimation problem from the
network/channel coding part, despite the fact that such a separation
may be suboptimal. Here we argue that the best model for the
distributed source code is one in which each encoder multicasts a
message to each subset of other nodes in the network, rather than
sending an individual message to each other node in the network.
To see that such a model is the appropriate one, consider a simple
wired network depicted in Fig. 2 in which three nodes (1,2,3)
making local observations Y(n)
1
,Y(n)
common underlying sequence T(n)would like to communicate over
the butterfly network in order to form local estimatesˆT(n)
of T(n). Because of the unidirectionality of the links, only node 1
may transmit information. Suppose further that the observations at
node 2 and 3 are statistically identical and the distortion metrics are
the same, and we wish to obtain the same target average estimation
error D2 = D3 at the two nodes. If node 1 encodes a separate
message for node 2 and node 3, then it would suffice to take these
two messages to be the same in this symmetric case. However, the
network code can not know this, because we have forced the source
coding construction to have a separate message for each of nodes 2
and 3. Thus, the network code is forced to attempt to transmit two
unicasts, one between 1 and 2 with rate R1→2, and one in between 1
and 3 with rate R1→3. If each link in the network is unit capacity, and
the network code is forced to treat the information flowing in between
nodes 1 and 2 as independently unicast from the unicast between 1
and 3, then the highest symmetric rate R = R1→2 = R1→3 which
can be obtained is 3/2. However, had we chosen our source code as
outputting three messages Q1→2,Q1→3,Q1→2,3, so that we included
one which was multicast from 1 to both 2 and 3, then the network
code could support a symmetric rate of R1→2,3 = 2 [10]. This would
not send any unicast information at all R1→2 = R1→3 = 0. This way
33% more useful information flows from 1 to 2 and 3 as would have
had we required only unicasts, and the distortion obtained at nodes
2
,Y(n)
3
statistically related to a
1
,ˆT(n)
2
,ˆT(n)
3
2 and 3 will thus be lower.
From this simple example we can easily infer that a proper sepa
rated source and network/channel coding approach treats the source
code within network node i as producing an array of 2M−1multicast
messages, with one message Qi→A for each subset A ⊆ [M] \ i.
The capabilities of the possible network/channel codes are then
summarized by a region C of vectors of such multicast rates
r := [Rj→Aj ∈ [M], A ⊆ [M] \ j]
which are simultaneously supportable by the network infrastructure.
The capabilities of the possible source codes are summarized by
a rate distortion region RD describing the set of simultaneously
achievable multicast rates r and average estimation errors
(1)
d := [Dii ∈ [M]],Di :=
1
N
N
?
n=1
E
?
di
?
T(n),ˆT(n)
i
??
(2)
An overall source channel code achieving average estimation errors
lower than d is selected by choosing a rate vector r that is in both
C and also in RD, i.e. with (r,d) ∈ RD. We now focus our efforts
on describing the rate distortion region for the associated family of
source codes we have selected.
III. ACHIEVABLE RATE DISTORTION REGION
The rate distortion region explains the relationship between the
length in bits of the different messages multicast between the nodes
and the estimation errors (measured in terms of average costs for
Bayesian estimation) that decoder/estimators at these nodes can
obtain. In particular, the vector (r,d) of multicast rates r :=
?Rj→A
N, encoders and decoders
??j ∈ [M],A ∈ 2[M]\j?
and average estimation errors d :=
[Djj ∈ [M]] is said to be achievable if there exists a block length
fN
j→A: YN
j → [LN
j→A],gN
i : YN
i ×
?
(j→A)∈Di
[LN
j→A] →ˆTi (3)
withˆTN
i
= gN
i(YN
i ,QDi) such that
Rj→A ≥
1
NlogLN
j→A,E
?
1
N
N
?
n=1
di(T(n),ˆT(n)
i
)
?
≤ Di
(4)
The rate distortion region RD for this problem is defined as the
closure of the region of achievable vectors (r,d).
Denotethesetofmessage
Si :=?(i → A)A ∈ 2[M]\i?
theorem.
indicesleavingnode
i
by
and the set {Ui→A  A ∈ 2[M]\i}
i∈[M]Si, then we have the followingas USi. If we define S :=?
Theorem1: Given a joint distribution pT,Y[M](t,y[M]), let
Ξ(d) be the collection of random vectors ξ = US which are jointly
distributed with T and Y[M]such that the following conditions are
satisfied
1) T,Y[M]\i,US\Si↔ Yi ↔ USifor all i ∈ [M]
2) There exists a decoding function gi : UDi×Yi →ˆTi such that
E [di(T,gi(UDi,Yi))] ≤ Di for all i ∈ [M]
For each ξ ∈ Ξ(d), define Φ(ξ) as in (5). Also, for each ξ ∈ Ξ(d)
and for each φ ∈ Φ(ξ), define RDin(ξ,φ) as in (6).
Let
RDin :=
ξ∈Ξ(d)
Then, the convex hull conv(RDin) of RDin is an inner bound to
the rate distortion region, i.e. conv(RDin) ⊆ RD.
??
φ∈Φ(ξ)
RDin(ξ,φ)
Page 3
Φ(ξ)=
?
?
˜RS :
?
?
(j→A)∈Pj
˜Rj→A >
?
?
(j→A)∈Pj
H(Uj→A) − H(UPjYj), ∀ Pj ⊆ Sj, j ∈ [M]
?
?
(5)
RDin(ξ,φ)=RS :
(j→A)∈Ci
Rj→A ≥
(j→A)∈Ci
˜Rj→A− H(Uj→A)
?
+ H(UCiUDi\Ci,Yi), ∀ Ci ⊆ Di, i ∈ [M]
?
(6)
Proof idea: This result is an adaptation of a well known inner
bound in the multiterminal source coding community known as the
BergerTung inner bound, as clarified by Han and Kobayashi [1], with
the twist that the multiple (dependent) descriptions at each encoder
require an additional set of encoder inequalities. A sketch of the proof
is provided in Appendix A. A more detailed proof may be found in
[11]. ?
We next analyze the structure of the achievable rate region,
because knowing the structure of the rate region may be helpful
when we optimize some function of rates over the rate region. We
indeed use some structural properties of the inner bound to simplify
our bound to simpler problems in Section IV, and, thus present
those structural properties below.
Proposition 1: For each ξ ∈ Ξ(d), Φ(ξ) is a contrapolymatroid.
Proof: The set S is implied to be the ground set, and the
rank function ρ : 2S→ R is defined as
ρ(P) ?
j∈[M]
We must show that ρ is indeed a rank function. Consider two sets Q
and P such that Q ⊆ P ⊆ S, then
ρ(P) − ρ(Q)
=H(Uj→A) − H(UP∩SjYj)
??
(j→A)∈P∩Sj
H(Uj→A) − H(UP∩SjYj)
(7)
?
j∈[M]
?
?
+H(UQ∩SjYj)
?
(j→A)∈Lj
?
=
?
0
j∈[M]
?
(j→A)∈Lj
H(Uj→A) − H(ULjUQ∩Sj,Yj)
?
≥
where Lj := (P ∩ Sj) \ (Q ∩ Sj). This establishes that ρ is non
decreasing. Next consider any two sets P ⊆ S and Q ⊆ S. We
have
ρ(P) + ρ(Q) − ρ(P ∩ Q) − ρ(P ∪ Q)
?
−H(UP∩SjYj) − H(UQ∩SjYj))
?
−H(UP∩Qc∩SjUP∩Q∩Sj,Yj)) ≤ 0
which implies that ρ is a rank function of a contrapolymatroid.
To see that this contrapolymatroid is equal to Φ(ξ), simply note
that evaluating the rank function ρ and writing the corresponding
inequality for every subset of Sj gives the list of inequalities for
node j. The collection of these inequalities over j ∈ [M] then
=
j∈[M]
(H(UP∩Q∩SjYj) + H(U(P∪Q)∩SjYj)
=
j∈[M]
(H(UP∩Qc∩SjUQ∩Sj,Yj)
yields Φ(ξ). Finally, note that evaluating the rank function at any
collection of indices corresponding to message sent from different
encoders simply sums the corresponding individual inequalities for
the different encoders. ?
Corollary 1: For each ξ ∈ Ξ(d), the generating vertices of
the polyhedron Φ(ξ) are exactly {φ(π)π ∈ Π(S)} where Π(S) is
the set of permutations of the indices in S, and φ(π) is the vector
given by
φπ(1)(π) ? ρ(π(1)) = I(Uπ(1);Y[M])
and for every i ∈ {2,...,S}
φπ(i)(π)
?
=
ρ({π(1),...,π(i)}) − ρ({π(1),...,π(i − 1)})
I(Uπ(i);U{π(1),...,π(i−1)},Y[M])
and where ρ is the rank function defined in (7). Additionally, for any
λ ∈ RS
is attained by φ(π) for π any permutation of the elements of S
such that λπ(1)≥ ··· ≥ λπ(S).
(8)
+, then the solution to the linear program minφ∈Φ(ξ)λ · φ
Proof: These are standard properties of contrapolymatroids.
See, for instance, Lemma 3.3 of [12]. ?
We next use these structural properties of the achievable rate
distortion region to simplify this bound to two simpler problems:
the multiple descriptions problem and the CEO problem.
IV. SIMPLIFICATION OF BOUNDS TO SIMPLER PROBLEMS
Because we have argued that the collaborative distributed estima
tion problem is essentially a hybrid between a collection of CEO
problems and a multiple descriptions problem, it is important to show
that the inner bound we have given specializes to known inner bounds
for these problems in special cases.
A. Simplification to Multiple Descriptions Problem
The multiple descriptions problem for two descriptions can be
obtained as a special case of our collaborative estimation problem
for M = 4 nodes. Only one node, say node 1, gets to make
observations which it would like to inform the other 3 network
nodes about, so that Y(n)
1
= T(n)and Y(n)
Additionally, node 1 structures its encodings so that nodes 2 and 3
receive different encodings, while node 4 receives everything that is
available to node 2 and 3. The coding strategy introduced in [6] to
this problem can be accomplished by dividing Q1→{4}up into two
parts Q1→{4}= (Q1
bits per symbol with ∆1+∆2 = R1→{4}and forming two descrip
tions X1 ? (Q1→{2,4},Q1
When only one of the two descriptions X1 or X2 is available, the
achievability coding strategy introduced in [6] simply discards the
part of the description associated with Q1→{4} and utilizes only
U1→{3,4} or U1→{2,4}, respectively. When both descriptions are
available, the achievability coding strategy introduced in [6] uses
all of the encodings (Q1→{2,4},Q1→{3,4},Q1→{4}). Additionally,
since R1 = R2,4+ ∆1 and R2 = R3,4+ ∆2, we can remove the
i
= 0 for all i ?= 1.
1→{4},Q2
1→{4}) containing ∆1 ≥ 0 and ∆2 ≥ 0
1→{4}) and X2 ? (Q1→{3,4},Q2
1→{4}).
Page 4
redundant variables ∆1 and ∆2, and rewrite the constraint for R4
as R4 = R1 − R2,4 + R2 − R3,4. These identifications may be
summarized with the following notation
U1→{2,3}? U2,3, U1→{3,4}? U3,4, U1→{4,}? U4
R1→{2,3}? R2,3, R1→{3,4}? R3,4, R1→{4,}? R4
˜R1→{2,3}?˜R2,3,˜R1→{3,4}?˜R3,4,˜R1→{4}?˜R4
Uj→A = ∅, Rj→A = ∅,˜Rj→A = ∅ all other A
Where the auxiliary random variables U4,U2,4,U3,4are selected such
that
p(U4,U2,4,U3,4,T) = p(T)p(U4,U2,4,U3,4T)
D1 ≥ E?d(T,ˆT2)?, D2 ≥ E?d(T,ˆT3)?, D0 ≥ E?d(T,ˆT4)?
Under these identifications, the inner bound becomes
(9)
(10)
(11)
(12)
(13)
R4
≥
≥
≥
˜R4− H(U4) + H(U4U3,4,U2,4)
˜R2,4
˜R3,4
(14)
(15)
(16)
R2,4
R3,4
Having the inequalities R1 ≥ R2,4, R2 ≥ R3,4 (because ∆1,∆2 ≥
0) in hand, we replace R4 with R1−R2,4+R2−R3,4 in (14) and
use the inequalities (14)(16) to obtain a bound on the rate region
(R1,R2) which is given by
R1
≥
≥
≥
˜R2,4
˜R3,4
˜R2,4+˜R3,4+˜R4
−H(U4) + H(U4U3,4,U2,4)
(17)
(18)
R2
R1+ R2
(19)
We note that the minimum of˜R2,4+˜R3,4+˜R4 from the encoder
inequalities to be
H(U4) + H(U2,4) + H(U3,4) − H(U4,U2,4,U3,4T)
Thus right hand side of (19) becomes
H(U2,4) + H(U3,4) − H(U4,U2,4,U3,4T)
+H(U4U2,4,U3,4)
H(U2,4) + H(U3,4) − H(U4,U2,4,U3,4T)
+H(U4,U2,4,U3,4) − H(U2,4,U3,4)
I(U2,4;U3,4) + I(T;U4,U2,4,U3,4)
=
=
We next point out that by the contrapolymatroid property of
the source encoder region describing the collection of variables
˜R2,4,˜R3,4,˜R4 by Corollary 1, this minimum is attained for 6
(permutations of λ1 = 1,λ2 = 1,λ3 = 1) possible solutions of
˜R2,4,˜R3,4,˜R4. However, we are interested in only two of the 6
solutions which are useful in finding the region of (R1,R2) and
present the values of˜R2,4,˜R3,4 below.
1) ˜R2,4 = I(U2,4;T),
˜R3,4 = I(U3,4;U2,4,T)
2) ˜R2,4 = I(U2,4;U3,4,T),
˜R3,4 = I(U3,4;T)
Using time sharing argument of these two solutions we write the
region of rates (R1,R2) as
R1
≥
≥
≥
I(U2,4;T) + α I(U2,4;U3,4T)
I(U3,4;T) + (1 − α) I(U2,4;U3,4T)
I(U2,4;U3,4) + I(T;U4,U2,4,U3,4)
(20)
(21)
(22)
R2
R1+ R2
where 0 ≤ α ≤ 1. We next show that any point in the achievable rate
region (ECG region) proved in [6] also lies in the region we proved
above. To prove this, we rewrite the EGC region in the following
form
r1
≥
≥
I(U2,4;T)
max{I(U3,4;T),
I(U2,4;U3,4) + I(T;U4,U2,4,U3,4) − r1}
?r1− I(U2,4;T)
r2
and let
α = min
I(U2,4;U3,4T),1
?
(23)
Then
R1 ≥ min{r1,I(U2,4;T) + I(U2,4;U3,4T)} ≤ r1
(24)
and
R2
≥
I(U3,4;T) + I(U2,4;U3,4T)
−min{r1− I(U2,4;T), I(U2,4;U3,4T)}
max{I(U3,4;T),
I(U2,4;T) + I(U3,4;T) + I(U2,4;U3,4T) − r1}
r2
=
≤
In the above proof we used the following inequality which can be
easily proved.
I(U2,4;U3,4) + I(T;U4,U2,4,U3,4)
I(U2,4;T) + I(U3,4;T) + I(U2,4;U3,4T)
This completes the proof that our inner bound contains every point
in the EGC region.
≥
B. Simplification to CEO problem
We next show that CEO problem can be obtained as a simplifi
cation of our model and that our inner bound simplifies the Berger
Tung inner bound for this case. To see this, suppose that the nodes
i ∈ [M] \ M observe the common phenomenon embodied by the
sequence T(n)and send one message each to the CEO node M.
Using these messages received from the nodes i ∈ [M −1], the CEO
node produces an estimateˆT (ˆTM =ˆT) of T such that the expected
distortion E[d(T,ˆT)] < D.
Since the nodes i ∈ [M −1] send messsages only to node M, we
set the rates corresponding to the other messages to 0 and redefine
the rates and variables relevent to this problem as follows.
Rj→M ? Rj, Uj→M ? Uj ∀ j ∈ [M − 1]
Rj→A = 0,˜Rj→A = 0,Uj→A = ∅ all other A,j ∈ [M − 1]
DM ? D, RM→A = 0,˜RM→A = 0,UM→A = ∅
Note that the random vectors ξ = (U[M−1]) satisfy the following
constraints.
• T,Y[M−1]\i,U[M−1]\j↔ Yj ↔ Uj for all j ∈ [M − 1]
• There exists a decoding function g : U[M−1] →ˆT such that
D > E?d(T,ˆT)?
Φ(ξ) = {˜RD˜Rj > H(Uj− H(UjYj),∀j ∈ [M − 1]}
Here,˜Rj can be selected such that˜Rj = I(Uj;Yj) + ?j for all
j ∈ [M − 1] where ?j can be made arbitrarily small. Note that
selecting the rates so will not change the rate region. If we select
˜Rj = I(Uj;Yj) + ?j, there will be only 1 rate vector˜RD in the set
If we denote the set [M −1] as D := [M −1] as, then Φ(ξ) becomes
Page 5
Φ(ξ). Thus, Ψ is only a function of ξ, i.e. Ψ(ξ,φ) = Ψ(ξ). Hence,
Ψ(ξ) is the collection of rate vectors RD ≥ 0 obeying
?
=H(UCUD\C) −
j∈C
Rj
>
?
j∈C
(˜Rj− H(Uj)) + H(UCUD\C)
?
H(UCUD\C) − H(UCYC)
H(UCUD\C) − H(UCYC,UD\C)
I(UC;YCUD\C)
j∈C
H(UjYj)
=
=
=
for all C ⊆ D. Here, we have used the facts that node M (CEO)
does not have any side information (YM = 0) and UC ↔ YC ↔
UD\C. Thus the inner bound for the ratedistortion region for the
CEO problem becomes
RDin =
(R[M−1],D)
??????
R[M−1]∈
?
ξ∈Ξ(D)
Ψ(ξ)
where Ξ(D) is the collection of random vectors ξ. This is exactly
the Berger Tung inner bound for the CEO problem given in [4].
C. Simplification to Side Information May be Absent case
The problem studied in [13] can also be obtained as a simplification
of our model. To see this, let the number of nodes M = 3 and,
suppose that node 3 directly observes the source, i.e. Y3 = T, and
node 1 has side information about the source Y1 = Y while node 2
has no side information. Also, suppose that node 3 sends a common
description to both 1,2 and an individual description to only node 1
as it is implicitly done in [13]. We can show that sum of the rates of
these two descriptions derived from our inner bound is equal to the
ratedistortion function proved for the sumrate in [13]. We skip the
proof to conserve the space and refer any interested reader to [11].
We can also show that the sum rate result proved for the general
problem with degraded side information can be retrieved from our
inner bound.
V. CONCLUSION
We analyzed optimized code constructions for collaborative dis
tributed estimation via multiterminal information theory. We argued
that the proper model for a distributed source code for collaborative
distributed estimation involves multiple multicast messages from
each encoder rather than unicast messages, yielding a hybrid coding
problem between multiple descriptions and the CEO problem. An
achievable rate region which hybridized the Berger Tung inner bound
and multiple descriptions proof techniques were presented. The inner
bound was shown to be equal to the known bounds for some simpler
problems by exploiting the structural properties of the rate region.
APPENDIX A
PROOF OF THEOREM
We present a sketch of the proof of the inner bound given in Section
III here.
Proof Select a joint conditional distribution p?uS
functions {gN
Calculate the marginal distributions p(uj→A).
Codebook Generation: At each node j ∈ [M], for each subset
of nodes A ⊆ 2[M]\j, generate a codebook with 2n˜
codewords by randomly drawing the elements such that they are i.i.d.
??t,y[M]
?, a set
of encoding functions {fN
i  i ∈ [M]} such that the rates RS are in RDin.
j→A (j → A) ∈ S } and a set of decoding
Rj→AlengthN
according to the distribution p(uj→A), where?
codewords by mj→A ∈ {1,...,2n˜
into 2nRj→Abins by randomly and uniformly assigning the indices
to the bins. Index the bins by bj→A ∈ {1,...,2nRj→A} and denote
the set of codewords in bin bj→A by Bj→A(bj→A).
Encoding: At each node j ∈ [M], encode the observation sequence
YN
j
by selecting one codeword UN
Cj→A, for each (j → A) ∈ Sj}, such that?UN
If there are more than one such UN
with the smallest indices under lexicographic ordering. If there is
no such UN
subset of nodes A ⊆ 2[M]\j, send the index bj→A of the bin that
contains UN
Bj→A(bj→A). This requires Rj→A bits to multicast a message to a
subset of nodes A ⊆ 2[M]\j.
Decoding: At each node i ∈ [M], decode the messages re
ceived at the node by selecting the codeword UN
Bj→A(bj→A) for each (j → A) ∈ Di such that?UN
a set of codewords, select an arbitrary set of codewords. Reproduce
the underlying sequence TNbyˆTN
(j→A)∈Pj
˜Rj→A >
?
(j→A)∈PjH(Uj→A) − H(UPjYj) for each Pj ⊆ Sj. Index the
Rj→A}. Partition the codewords
j→A(mj→A) from each codebook
Sj(mSj), YN
? is the set of strongly typical sequences.
Sj(mSj), select the codewords
j
?
∈
A∗
?(USj,Yj), where A∗
Sj(mSj), select an arbitrary set of codewords. For each
j→A(mj→A) to the nodes in A, i.e. UN
j→A(mj→A) ∈
j→A(?j→A) in bin
Di(?Di),YN
i
?
∈
A∗
?(UDi,Yi), where UDi? (Uj→A)(j→A)∈Di. If there is no such
i
= gN
i
?YN
i ,UN
Di(?Di)?. ?
REFERENCES
[1] T. S. Han and K. Kobayashi, “A unified achievable rate region for a gen
eral class of multiterminal source coding systems,” IEEE Transactions
on Information Theory, vol. IT26, no. 3, pp. 277–288, May 1980.
[2] A. B. Wagner and V. Anantharam, “An improved outer bound for
multiterminal source coding,” IEEE Transactions on Information Theory,
vol. 54, no. 5, pp. 1919–1937, May 2008.
[3] T. Berger, Z. Zhang, and H. Viswanathan, “The ceo problem,” IEEE
Transactions on Information Theory, vol. 42, no. 3, pp. 887–902, May
1996.
[4] J. Chen, X. Zhang, T. Berger, and S. B. Wicker, “An upper bound on
the sumrate distortion function and its corresponding rate allocation
schemes for the ceo problem,” IEEE Journal on Selected Areas in
Communications, vol. 22, no. 6, pp. 977–987, August 2004.
[5] Y. Oohama, “Gaussian multiterminal source coding,” IEEE Transactions
on Information Theory, vol. IT43, no. 6, pp. 1912–1923, November
1997.
[6] A. A. El Gamal and T. M. Cover, “Achievable rates for multiple
descriptions,” IEEE Transactions on Information Theory, vol. IT28,
no. 6, pp. 851–857, November 1982.
[7] S. C. Draper and G. W. Wornell, “Side information aware coding
strategies for sensor networks,” IEEE Journal on Selected Areas in
Communications, vol. 22, no. 6, pp. 966–976, August 2004.
[8] A. D. Wyner and J. Ziv, “The ratedistortion function for source coding
with side information at the decoder,” IEEE Transactions on Information
Theory, vol. IT22, no. 1, pp. 1–10, January 1976.
[9] Y. Oohama, “Ratedistortion theory for gaussian multiterminal source
coding systems with several side informations at the decoder,” IEEE
Transactions on Information Theory, vol. 51, no. 7, pp. 2577–2593, July
2005.
[10] S.Y. R. Li, R. W. Yeung, and N. Cai, “Linear network coding,”
IEEE Transactions on Information Theory, vol. 49, no. 2, pp. 371–381,
February 2003.
[11] J. M. Walsh and S. Ramanan, “Coding perspectives for collaborative
distributed estimation over networks,” proof of inner bound. [Online].
Available: http://www.ece.drexel.edu/walsh/web/listOfPubs.html
[12] D. N. C. Tse and S. V. Hanly, “Multiaccess fading channels  part
i: Polymatroid structure, optimal resource allocation and throughput
capacities,” IEEE Transactions on Information Theory, vol. 44, no. 7,
pp. 2796–2815, November 1998.
[13] C. Heegard and T. Berger, “Rate distortion when side information may
be absent,” IEEE Transactions on Information Theory, vol. 31, no. 6,
pp. 727–734, November 1985.