Page 1

arXiv:0809.3540v1 [cond-mat.stat-mech] 22 Sep 2008

A Note on the Equivalence of Gibbs Free Energy and Information

Theoretic Capacity

David Ford, Physics NPS∗

Department of Physics, Naval Post Graduate School, Monterey, California

(Dated: September 22, 2008)

The minimization of Gibbs free energy is based on the changes in work and free energy that occur

in a physical or chemical system. The maximization of mutual information, the capacity, of a noisy

channel is determined based on the marginal probabilities and conditional entropies associated with a

communications system. As different as the procedures might first appear, through the exploration

of a simple, “dual use” Ising model, it is seen that the two concepts are in fact the same. In

particular, the case of a binary symmetric channel is calculated in detail.

INTRODUCTION

In 1876 J.W. Gibbs [1] proposed a refinement

to the notion of Helmholtz free energy. The util-

ity of the refinement rests on the observation

that when attempting to determine the max-

imum amount of work that may be extracted

from a thermodynamic process, some of that

work may already be accounted for, for exam-

ple, work against the atmosphere −p∆V . The

suggestion was that perhaps the actual quan-

tity of interest is the maximum amount of work

other than atmospheric that may be extracted.

This general concept, equipped with a system

dependent form for the work term [2],[3],[4],[5] is

well known and widely used in physics, physical

chemistry and many engineering disciplines.

Some seventy years later C. Shannon and W.

Weaver [6] provide a sound theoretical frame-

work for determining the maximum amount

of information that may be transmitted from

sender to receiver through a noisy channel.

Their concept, the channel capacity, calculated

by extremization of differences of various com-

munication system entropies, is reminiscent of

statistical physics [7] but apparently different[8].

The purpose of this letter is to show how these

two concepts are in fact the same. To accom-

plish this it is useful to have a simple example

that belongs to both the physics and communi-

cations theory traditions. The Ising model for

magnetic systems is a good candidate. The liter-

ature on the topic is vast. Physical applications

of the model relevant to the present purpose in-

clude [9], [10],[11]. Relevant applications of the

model in communications include [12],[13],[14].

After a brief review of the Ising model, it will be

shown that the real space renormalization pro-

cedure [15] connects the Gibbs free energy and

the channel capacity in a natural way.

REVIEW

As noted in the introduction, the Ising model

is of interest in both physics and communica-

tions. The states described by the model are a

sequence of electron spin up (↑) or spin down

(↓) states which are easily translated to the 1’s

and 0’s of a bit stream. The simplest communi-

cations scheme is one in which the transmitted

bits are statistically independent of each other

but coupled to the bits received. The sequence

FIG. 1: A simple send/receive Ising model. The sent

bits, σ1, σ3, σ3 (odd numbered) are independent of

each other but coupled to their received bits σ2, σ4,

σ6 (even numbered).

Page 2

2

shown in figure 1 consists of six sites represent-

ing 26possible configurations of spins (bits).

Following the paradigm of Boltzmann and

Gibbs, the probability weight of each spin con-

figuration is a function of its own particular en-

ergy as well as the energy of its environment,

represented by the temperature [16], [17]. Al-

though at first the pedagogy may seem spe-

cialized to a narrow class of systems occurring

in physics, engineering and chemistry, the ap-

proach is quite general and any discrete prob-

ability distribution {p1,p2,...,pN} can be put

into Boltzmann-Gibbs form [18].

The assumed statistical independence of the

send/receive pairs manifests itself in terms of the

physics through the form of the energy function.

Under these conditions, the energy of a config-

uration, Hσ1,...,σ6, is comprised of additive con-

tributions from the distinct send/receive pairs

with no interaction energy terms

Hσ1,...,σ6= Hσ1σ2+ Hσ3σ4+ Hσ5σ6.

For this reason, to understand the properties

of the total system, it is enough to study the

properties of a single send/receive pair. A typ-

ical Ising Hamiltonian, appropriate for a single

send/receive unit, for example site 1 (send) and

site 2 (receive) as shown in figure 1, is of the

form

Hσ1σ2= −Jσ1σ2+ h(σ1+ σ2) + c

with spin coupling J and external field h. The

so-called Zeeman energy term, c, is initially zero.

Later, as a result of the decimation, the con-

straints imposed by the channel and energy min-

imization, it will be assigned a value.

The four possible states of a basic unit are

listed in figure 2. For a concrete example, H↑↓

is computed as

−J(1 ∗ −1) + h(1 + −1) = J.

The remaining energies are computed similarly.

The following conventions and definitions are

standard and will be referred to in the sequel.

Let β, defined as the inverse of the send/receive

FIG. 2: The possible states of the basic unit (one

send, one receive) and corresponding numerical val-

ues.

unit system temperature, be constant (isother-

mal). The Helmholtz free energy of the system,

F, is defined by

Z = e−βF

where the partition function (the sum over all

configurations) is given by

Z = e−βH↑↑+ e−βH↑↓+ e−βH↓↑+ e−βH↓↓.

The probability of any particular spin configura-

tion, for example ↑↑ (send a “1”, receive a “1”)

is represented in Boltzmann Gibbs form as

P↑↑=e−βH↑↑

Z

.

The concept of internal energy, denoted U, is

a macroscopic observable and a common object

of study in thermodynamics. Changes in the in-

ternal energy , ∆U, are decomposed into two

distinct macroscopic types: work ∆W and heat

∆Q. As noted in the introduction, macroscopic

descriptions of work are varied and system de-

pendent. Thermodynamic expressions of Gibbs

free energy often inherit this system dependent

Page 3

3

language. Statistical mechanics offers an addi-

tional point of view. In terms of the microscopic

data, the internal energy associated with a ba-

sic send/receive unit is also well represented by

[16], [17]

U = P↑↑H↑↑+ P↑↓H↑↓+ P↓↑H↓↑+ P↓↓H↓↓.

Similarly, the generic form of work in terms of

given changes in the microscopic data is [18]

∆W = P↑↑∆H↑↑+ P↑↓∆H↑↓+

P↓↑∆H↓↑+ P↓↓∆H↓↓.

A TRIVIAL DECIMATION

A channel is represented mathematically as a

matrix of conditional probabilities. For exam-

ple, the probability that the receiver observes a

↓ given that a ↑ was sent: P↓|↑. The channel for

the send/receive unit system introduced in the

last section is simply

?P↑|↑ P↓|↑

P↑|↓ P↓|↓

?

where the rows of the matrix sum to one. In

terms of the Boltzmann Gibbs distribution, the

probability of receiving a spin up given that a

spin up was sent is

P↑|↑=

e−βH↑↑

e−βH↑↑+ e−βH↑↓.

The probability of receiving a spin down given

that a spin up was sent is

P↓|↑=

e−βH↑↓

e−βH↑↑+ e−βH↑↓.

Define the standard [15] rescaled Hamiltonian,

˜H, by summing out (decimation) the spin values

at site 2 (the “receiver”)

e−β˜ H↑= e−βH↑↑+ e−βH↑↓

so that by the definition of the channel

˜H↑=1

βLog(P↑|↑) + H↑↑

or equivalently

˜H↑=1

βLog(P↓|↑) + H↑↓.

In the parlance of communications theory, the

“sender’s hamiltonian”,

Boltzmann Gibbs) the “sender’s marginal”,˜P.

Notice that the channel is invariant under a

transformation

˜H, determines (via

H↑↑→ H↑↑+ c↑

˜H↑ →˜H↑ + c↑.

In a similar way,˜H↓and c↓are obtained in terms

of the original unit system and channel via dec-

imation.

A single constant is sufficient to arbitrarily

adjust the sender’s marginal at fixed channel.

Without loss of generality set

c↓= −c↑.

Interestingly, from a pedagogical point of view,

this is the only choice consistent with the as-

sumption of a canonically distributed unit sys-

tem [19]. The renormalized (sender’s) partition

function,˜Z, is defined

˜Z = e−β˜ H↑+ e−β˜ H↓.

Some Quantities of Physical Interest

The capacity is defined as the (maximized)

difference of two information theoretic quanti-

ties: the entropy associated with the receiver’s

marginal distribution, S(ˆP) and the average of

the channel entropies according to the sender,

< Schannel>˜ P. But what is the connection to

the physics?

Consider first < Schannel >˜ P. This is sim-

ply (−β times) the work done on the four state

system by the˜H decimation. For example

˜P↑

?−P↑|↑Log(P↑|↑) − P↓|↑Log(P↓|↑)?

Page 4

4

is easily seen to be

−β(P↑↑(˜H↑− H↑↑) + P↑↓(˜H↑− H↑↓))

with the˜P↓terms making a similar contribution.

Secondly, the receiver’s entropy, S(ˆP) is cal-

culated in terms of the sender’s marginal˜P and

the channel columns as

−

?

P↑|↑

P↑|↓

?

·˜ P Log

??

P↑|↑

P↑|↓

?

·˜ P

?

−

?

P↓|↑

P↓|↓

?

·˜ P Log

??

P↓|↑

P↓|↓

?

·˜ P

?

.

Exercise the right to decimate the system over

the spin at site 1 and defineˆH

e−βˆ H↑= e−βH↑↑+ e−βH↓↑

e−βˆ H↓= e−βH↑↓+ e−βH↓↓.

The entropy of the receiver’s marginal may then

be expressed

β(P↑↑ˆH↑+P↓↑ˆH↑+P↑↓ˆH↓+P↓↓ˆH↓+1

βLogZ).

Putting the pieces together, it is seen that the

channel capacity, i.e. the (maximized) quantity

S(ˆP)− < Schannel>˜ P

is given by the maximum over c↑of

β

?

spins

P••([˜H•+ˆH•] − H••) + LogZ.

(1)

This is easily seen to be (−β times) the change

in the Gibbs free energy

∆G = −∆W + ∆F

where the change in work, −∆W (for a fa-

mous, macroscopic example, −(−pdV )) and the

change in Helmholtz free energy, ∆F, are asso-

ciated with separating a composite microscopic

spin configuration, for example ↑↓, into its com-

ponent spins, ↑ and ↓, via decimations. By in-

spection, the work term, ∆W, in equation (1) is

apparent in its microscopic, statistical mechan-

ical form “p∆H”. A moments reflection identi-

fies the free energy changes. For by the defini-

tion of the decimated partition functions˜Z and

ˆZ, the change in the Helmholtz free energy, ∆F,

is given by

−1

β([Log˜Z + LogˆZ] − LogZ) = −1

βLogZ.

In this way it is seen that determining the ca-

pacity of the two site spin system is equivalent

to the standard Gibbs free energy analysis. The

capacity is the maximum amount of work (other

than the work done by the sender/receiver deci-

mations) that may be extracted from the system

consistent with the channel constraints.

Symmetric Channel

The prototypical example of a simple commu-

nications system is the binary symmetric chan-

nel. In the context of the Ising prototype dis-

cussed in this letter, the channel symmetry is

achieved by setting the applied field h equal to

zero. The hamiltonian

Hσ1σ2= −Jσ1σ2

depends on the coupling term J and

?P↑|↑ P↓|↑

P↑|↓ P↓|↓

?

=

?1 − δδ

δ

1 − δ

?

where

δ =

e−βJ

e−βJ+ eβJ.

As described in the previous section, decimating

over the “receivers spin” defines the˜H hamilto-

nian

˜H↑= −1

˜H↓= −1

βLog?e−βJ+ eβJ?+ c↑

βLog?e−βJ+ eβJ?− c↑.

Recall that the inclusion of c↑leaves the channel

invariant and is sufficient to arbitrarily adjust˜P,

the senders marginal distribution.

ObtainˆH similarly, i.e. by decimation over

the “senders spin” and respecting the channel

constraints

ˆH↑= −1

ˆH↓= −1

βLog?e−β(J−c↑)+ eβ(J−c↑)?

βLog?e−β(J+c↑)+ eβ(J+c↑)?.

Page 5

5

With these quantities in hand, the Gibbs free

energy changes consistent with the channel con-

straints may be computed, following equation

(1), as a function of c↑. Figure 3 shows the re-

sults for the case β = 1, J = 1. With these

parameter settings and accompanying channel

constraints, the minimum ∆G, i.e.

the channel capacity, is achieved at c↑= 0.

−1 times

FIG. 3: Upper frame: the work and free energy

changes associated with separating the composite

spin states into components at β = 1, J = 1. Lower

frame: the changes in Gibbs free energy as c↑is var-

ied, i.e. extremization over the sender’s marginal

distribution.

SUMMARY

The binary channel is simultaneously a basic

example of a communications system often stud-

ied in information theory and a basic example of

a magnetic spin system. A straightforward com-

parison of the associated channel capacity and

Gibbs free energy procedures finds no difference

(other than multiplication by -β) between the

two concepts.

BIBLIOGRAPHY

∗dkford@nps.edu

[1] J.W. Gibbs, “On the Equilibrium of Hetero-

geneous Substances”, Transactions of the Con-

necticut Academy of Arts and Sciences 3, 1876

[2] H. Callen, “Thermodynamics and an Introduc-

tion to Thermostatistics”, Wiley, 1985

[3] L. Landau; E. Lifshitz, “Theory of Elasticity”,

Pergamon, Oxford, 1986

[4] B. Krevet, M. Kohl, P. Morrison, S. Seelecke,

“Magnetization and strain dependent free en-

ergy model for FEM simulation of magnetic

shape memory alloys”, Eur. Phys. J. Special

Topics, 158, 2008

[5] S.J. Kim, “A rate dependent thermo-electro-

mechanical free energy model for perovskite

type single crystals”, International Journal of

Engineering Science, 45, 2007

[6] C. Shannon; W. Weaver, “The Mathematical

Theory of Communication”, University of Illi-

nois Press, 1949

[7] E.T. Jaynes, “Information Theory and Statis-

tical Mechanics”, Phys. Rev., 106, 620, 1957

[8] R. Feynman, M. Sands, “The Feynman Lec-

tures on Physics, vol. 3”, Addison Wesley, 1998

[9] T. Burkhardt; W. Kinzel, “Real-space renor-

malization for structural phase transitions in

the Ising universality class”, Physical Review

B, Volume 20, Issue 11, 1979

[10] Chin-Kun Hu, Hong-Yuh L “Renormalization-

Group Study of a Simple Cubic Ising Model

with Four-Spin Interaction”, Chinese Journal

of Physics, Volume 22, No. 3, 1984

[11] J. Fortin, P. Holdsworth, “ Real space renor-

malization group analysis of the random field

Ising model”, J. Phys. A: Math. Gen. 29 L539-

L545, 1996

[12] H. Nishimori, “ Optimum Decoding Tempera-

ture for Error-Correcting Codes”, J. Phys. Soc.

Jpn. 62 pp. 2973-2975, 1993

[13] H. Nishimori, K. Wong, “ Statistical mechan-

ics of image restoration and error-correcting

codes”, Phys. Rev. E 60, 132 - 144, 1999

[14] T. Tanaka,“Statistical mechanics of CDMA

multiuser demodulation”,

vol. 54, pp. 540546, 2001

[15] L. Kadanoff, “ Statistical Physics, Statics, Dy-

namics and Renormalization”, World Scientific,

2000

Euro- phys. Lett.,