Data compression at its base is concerned with how information is organized in data. Understanding this organization can lead to efficient ways of representing the information and hence data compression. In this paper we review the ways in which ideas and approaches fundamental to the theory and practice of data compression have been used in the area of bioinformatics. We look at how basic theoretical ideas from data compression, such as the notions of entropy, mutual information, and complexity have been used for analyzing biological sequences in order to discover hidden patterns, infer phylogenetic relationships between organisms and study viral populations. Finally, we look at how inferred grammars for biological sequences have been used to uncover structure in biological sequences.
This paper describes a free energy principle that tries to explain the ability of biological systems to resist a natural tendency to disorder. It appeals to circular causality of the sort found in synergetic formulations of self-organization (e.g., the slaving principle) and models of coupled dynamical systems, using nonlinear Fokker Planck equations. Here, circular causality is induced by separating the states of a random dynamical system into external and internal states, where external states are subject to random fluctuations and internal states are not. This reduces the problem to finding some (deterministic) dynamics of the internal states that ensure the system visits a limited number of external states; in other words, the measure of its (random) attracting set, or the Shannon entropy of the external states is small. We motivate a solution using a principle of least action based on variational free energy (from statistical physics) and establish the conditions under which it is formally equivalent to the information bottleneck method. This approach has proved useful in understanding the functional architecture of the brain. The generality of variational free energy minimisation and corresponding information theoretic formulations may speak to interesting applications beyond the neurosciences; e.g., in molecular or evolutionary biology.
Biosemiotics and cybernetics are closely related, yet they are separated by the boundary between life and non-life: biosemiotics is focused on living organisms, whereas cybernetics is applied mostly to non-living artificial devices. However, both classes of systems are agents that perform functions necessary for reaching their goals. I propose to shift the focus of biosemiotics from living organisms to agents in general, which all belong to a pragmasphere or functional universe. Agents should be considered in the context of their hierarchy and origin because their semiosis can be inherited or induced by higher-level agents. To preserve and disseminate their functions, agents use functional information - a set of signs that encode and control their functions. It includes stable memory signs, transient messengers, and natural signs. The origin and evolution of functional information is discussed in terms of transitions between vegetative, animal, and social levels of semiosis, defined by Kull. Vegetative semiosis differs substantially from higher levels of semiosis, because signs are recognized and interpreted via direct code-based matching and are not associated with ideal representations of objects. Thus, I consider a separate classification of signs at the vegetative level that includes proto-icons, proto-indexes, and proto-symbols. Animal and social semiosis are based on classification, and modeling of objects, which represent the knowledge of agents about their body (Innenwelt) and environment (Umwelt).
A mobile loop changes its conformation from "open" (free enzyme) to "closed" upon ligand binding. The difference in the Helmholtz free energy, ΔF(loop) between these states sheds light on the mechanism of binding. With our "hypothetical scanning molecular dynamics" (HSMD-TI) method ΔF(loop) = F(free) - F(bound) where F(free) and F(bound) are calculated from two MD samples of the free and bound loop states; the contribution of water is obtained by a thermodynamic integration (TI) procedure. In previous work the free and bound loop structures were both attached to the same "template" which was "cut" from the crystal structure of the free protein. Our results for loop 287-290 of AcetylCholineEsterase agree with the experiment, ΔF(loop)~ -4 kcal/mol if the density of the TIP3P water molecules capping the loop is close to that of bulk water, i.e., N(water) = 140 - 180 waters in a sphere of a 18 Å radius. Here we calculate ΔF(loop) for the more realistic case, where two templates are "cut" from the crystal structures, 2dfp.pdb (bound) and 2ace.pdb (free), where N(water) = 40 - 160; this requires adding a computationally more demanding (second) TI procedure. While the results for N(water) ≤ 140 are computationally sound, ΔF(loop) is always positive (18 ± 2 kcal/mol for N(water) = 140). These (disagreeing) results are attributed to the large average B-factor, 41.6 of 2dfp (23.4 Å(2) for 2ace). While this conformational uncertainty is an inherent difficulty, the (unstable) results for N(water) = 160 suggest that it might be alleviated by applying different (initial) structural optimizations to each template.
We present a novel analytical method to calculate conformational entropy of ideal cross-linking polymers from the configuration integral by employing a Mayer series expansion. Mayer-functions describing chemical bonds within the chain and for cross-links are sharply peaked over the temperature range of interest, and, are well approximated as statistically weighted Dirac delta-functions that enforce distance constraints. All geometrical deformations consistent with a set of distance constraints are integrated over. Exact results for a contiguous series of connected loops are employed to substantiate the validity of a previous phenomenological distance constraint model that describes protein thermodynamics successfully based on network rigidity.
The differential Shannon entropy of information theory can change under a change of variables (coordinates), but the thermodynamic entropy of a physical system must be invariant under such a change. This difference is puzzling, because the Shannon and Gibbs entropies have the same functional form. We show that a canonical change of variables can, indeed, alter the spatial component of the thermodynamic entropy just as it alters the differential Shannon entropy. However, there is also a momentum part of the entropy, which turns out to undergo an equal and opposite change when the coordinates are transformed, so that the total thermodynamic entropy remains invariant. We furthermore show how one may correctly write the change in total entropy for an isothermal physical process in any set of spatial coordinates.
The maximum entropy method is a theoretically sound approach to construct an analytical form for the probability density function (pdf) given a sample of random events. In practice, numerical methods employed to determine the appropriate Lagrange multipliers associated with a set of moments are generally unstable in the presence of noise due to limited sampling. A robust method is presented that always returns the best pdf, where tradeoff in smoothing a highly varying function due to noise can be controlled. An unconventional adaptive simulated annealing technique, called funnel diffusion, determines expansion coefficients for Chebyshev polynomials in the exponential function.
We study the global probability distribution of energy consumption per capita
around the world using data from the U.S. Energy Information Administration
(EIA) for 1980-2010. We find that the Lorenz curves have moved up during this
time period, and the Gini coefficient G has decreased from 0.66 in 1980 to 0.55
in 2010, indicating a decrease in inequality. The global probability
distribution of energy consumption per capita in 2010 is close to the
exponential distribution with G=0.5. We attribute this result to the
globalization of the world economy, which mixes the world and brings it closer
to the state of maximal entropy. We argue that global energy production is a
limited resource that is partitioned among the world population. The most
probable partition is the one that maximizes entropy, thus resulting in the
exponential distribution function. A consequence of the latter is the law of
1/3: the top 1/3 of the world population consumes 2/3 of produced energy. We
also find similar results for the global probability distribution of CO2
emissions per capita.
We develop the general reconstruction scheme in two scalar model. The quintom-like theory which may describe (different) non-singular Little Rip or de Sitter cosmology is reconstructed. The number of scalar phantom dark energy models (with Little Rip cosmology or asymptotically de Sitter evolution) is presented. Stability issue of such dark energy cosmologies as well as the flow to fixed points are studied. The stability of Little Rip universe which leads to dissolution of bound objects sometime in future indicates that no classical transition to de Sitter space occurs. The possibility of unification of inflation with Little Rip dark energy in two scalar theory is briefly mentioned.
We review a selection of methods for performing enhanced sampling in
molecular dynamics simulations. We consider methods based on collective
variable biasing and on tempering, and offer both historical and contemporary
perspectives. In collective-variable biasing, we first discuss methods stemming
from thermodynamic integration that use mean force biasing, including the
adaptive biasing force algorithm and temperature acceleration. We then turn to
methods that use bias potentials, including umbrella sampling and metadynamics.
We next consider parallel tempering and replica-exchange methods. We conclude
with a brief presentation of some combination methods.
This first article of a series formulates the thermodynamics of ideal gases
in a constant gravitational field in terms of an action principle that is
closely integrated with thermodynamics. The theory, in its simplest form, does
not deviate from standard practice, but it lays the foundations for a more
systematic approach to the various extensions, such as the incorporation of
radiation, the consideration of mixtures and the integration with General
Relativity. We study the interaction between an ideal gas and the photon gas,
and propose a new approach to this problem. We study the propagation of sound
in a vertical, isothermal column and are led to suggest that the theory is
incomplete, and to ask whether the true equilibrium state of an ideal gas may
turn out be adiabatic, in which case the role of solar radiation is merely to
compensate for the loss of energy by radiation into the cosmos. An experiment
with a centrifuge is proposed, to determine the influence of gravitation on the
equilibrium distribution with a very high degree of precision.
A convenient model for estimating the total entropy ({\Sigma}Si) of
atmospheric gases based on physical action is proposed. This realistic approach
is fully consistent with statistical mechanics, but uses the properties of
translational, rotational and vibrational action to partition the entropy. When
all sources of action are computed as appropriate non-linear functions, the
total input of thermal energy ({\Sigma}SiT) required to sustain a chemical
system at specific temperatures (T) and pressures (p) can be estimated,
yielding results in close agreement with published experimental third law
values. Thermodynamic properties of gases including enthalpy, Gibbs energy and
Helmholtz energy can be easily calculated from simple molecular and physical
properties. We propose that these values for entropy are employed both
chemically for reactions and physically for computing atmospheric profiles, the
latter based on steady state heat flow equilibrating thermodynamics with
gravity. We also predict that this application of action thermodynamics may
soon provide superior understanding of reaction rate theory, morphogenesis and
emergent or self-organising properties of many natural or evolving systems.
Searching for the dynamical foundations of the
Havrda-Charv\'{a}t/Dar\'{o}czy/Cressie-Read/Tsallis non-additive entropy, we
come across a covariant quantity called, alternatively, a generalized Ricci
curvature, an $N$-Ricci curvature or a Bakry-\'{E}mery-Ricci curvature in the
configuration/phase space of a system. We explore some of the implications of
this tensor and its associated curvature and present a connection with the
non-additive entropy under investigation. We present an isoperimetric
interpretation of the non-extensive parameter and comment on further features
of the system that can be probed through this tensor.
Since euclidean global AdS_2 space represented as a strip has two boundaries,
the state / operator correspondence in the dual CFT_1 reduces to the standard
map from the operators acting on a single copy of the Hilbert space to states
in the tensor product of two copies of the Hilbert space. Using this picture we
argue that the corresponding states in the dual string theory living on AdS_2 x
K are described by twisted version of the Hartle-Hawking states, the twists
being generated by a large unitary group of symmetries that this string theory
must possess. This formalism makes natural the dual interpretation of the black
hole entropy, -- as the logarithm of the degeneracy of ground states of the
quantum mechanics describing the low energy dynamics of the black hole, and
also as an entanglement entropy between the two copies of the same quantum
theory living on the two boundaries of global AdS_2 separated by the event
horizon.
Complex systems are often inherently non-ergodic and non-Markovian for which
Shannon entropy loses its applicability. In particular accelerating,
path-dependent, and aging random walks offer an intuitive picture for these
non-ergodic and non-Markovian systems. It was shown that the entropy of
non-ergodic systems can still be derived from three of the Shannon-Khinchin
axioms, and by violating the fourth -- the so-called composition axiom. The
corresponding entropy is of the form $S_{c,d} \sim \sum_i \Gamma(1+d,1-c\ln
p_i)$ and depends on two system-specific scaling exponents, $c$ and $d$. This
entropy contains many recently proposed entropy functionals as special cases,
including Shannon and Tsallis entropy. It was shown that this entropy is
relevant for a special class of non-Markovian random walks. In this work we
generalize these walks to a much wider class of stochastic systems that can be
characterized as `aging' systems. These are systems whose transition rates
between states are path- and time-dependent. We show that for particular aging
walks $S_{c,d}$ is again the correct extensive entropy. Before the central part
of the paper we review the concept of $(c,d)$-entropy in a self-contained way.
We study the information rates of non-coherent, stationary, Gaussian,
multiple-input multiple-output (MIMO) flat-fading channels that are achievable
with nearest neighbor decoding and pilot-aided channel estimation. In
particular, we investigate the behavior of these achievable rates in the limit
as the signal- to-noise ratio (SNR) tends to infinity by analyzing the capacity
pre-log, which is defined as the limiting ratio of the capacity to the
logarithm of the SNR as the SNR tends to infinity. We demonstrate that a scheme
estimating the channel using pilot symbols and detecting the message using
nearest neighbor decoding (while assuming that the channel estimation is
perfect) essentially achieves the capacity pre-log of non-coherent
multiple-input single-output flat-fading channels, and it essentially achieves
the best so far known lower bound on the capacity pre-log of non-coherent MIMO
flat-fading channels. We then extend our analysis to the multiple-access
channel.
The dynamics of dissipative fluids in Eulerian variables may be derived from an algebra of Leibniz brackets of observables, the metriplectic algebra, that extends the Poisson algebra of the frictionless limit of the system via a symmetric semidefinite component, encoding dissipative forces. The metriplectic algebra includes the conserved total Hamiltonian H, generating the non-dissipative part of dynamics, and the entropy S of those microscopic degrees of freedom draining energy irreversibly, which generates dissipation. This S is a Casimir invariant of the Poisson algebra to which the metriplectic algebra reduces in the frictionless limit. The role of S is as paramount as that of H, but this fact may be underestimated in the Eulerian formulation because S is not the only Casimir of the symplectic non-canonical part of the algebra. Instead, when the dynamics of the non-ideal fluid is written through the parcel variables of the Lagrangian formulation, the fact that entropy is symplectically invariant clearly appears to be related to its dependence on the microscopic degrees of freedom of the fluid, that are themselves in involution with the position and momentum of the parcel.
Starting from a very general trace-form entropy, we introduce a pair of
algebraic structures endowed by a generalized sum and a generalized product.
These algebras form, respectively, two Abelian fields in the realm of the
complex numbers isomorphic each other. We specify our results to several
entropic forms related to distributions recurrently observed in social,
economical, biological and physical systems including the stretched
exponential, the power-law and the interpolating Bosons-Fermions distributions.
Some potential applications in the study of complex systems are advanced.
Information content and compression are tightly related concepts that can be addressed through both classical and algorithmic information theories, on the basis of Shannon entropy and Kolmogorov complexity, respectively. The definition of several entities in Kolmogorov’s framework relies upon ideas from classical information theory, and these two approaches share many common traits. In this work, we expand the relations between these two frameworks by introducing algorithmic cross-complexity and relative complexity, counterparts of the cross-entropy and relative entropy (or Kullback-Leibler divergence) found in Shannon’s framework. We define the cross-complexity of an object x with respect to another object y as the amount of computational resources needed to specify x in terms of y, and the complexity of x related to y as the compression power which is lost when adopting such a description for x, compared to the shortest representation of x. Properties of analogous quantities in classical information theory hold for these new concepts. As these notions are incomputable, a suitable approximation based upon data compression is derived to enable the application to real data, yielding a divergence measure applicable to any pair of strings. Example applications are outlined, involving authorship attribution and satellite image classification, as well as a comparison to similar established techniques.
E.T. Jaynes, originator of the maximum entropy interpretation of statistical
mechanics, emphasized that there is an inevitable trade-off between the
conflicting requirements of robustness and accuracy for any inferencing
algorithm. This is because robustness requires discarding of information in
order to reduce the sensitivity to outliers. The principal of nonlinear
statistical coupling, which is an interpretation of the Tsallis entropy
generalization, can be used to quantify this trade-off. The coupled-surprisal,
-ln_k (p)=-(p^k-1)/k, is a generalization of Shannon surprisal or the
logarithmic scoring rule, given a forecast p of a true event by an inferencing
algorithm. The coupling parameter k=1-q, where q is the Tsallis entropy index,
is the degree of nonlinear coupling between statistical states. Positive
(negative) values of nonlinear coupling decrease (increase) the surprisal
information metric and thereby biases the risk in favor of decisive (robust)
algorithms relative to the Shannon surprisal (k=0). We show that translating
the average coupled-surprisal to an effective probability is equivalent to
using the generalized mean of the true event probabilities as a scoring rule.
The metric is used to assess the robustness, accuracy, and decisiveness of a
fusion algorithm. We use a two-parameter fusion algorithm to combine input
probabilities from N sources. The generalized mean parameter 'alpha' varies the
degree of smoothing and raising to a power N^beta with beta between 0 and 1
provides a model of correlation.
In this paper, we review and extend a family of log-det divergences for
symmetric positive definite (SPD) matrices and discuss their fundamental
properties. We show how to generate from parameterized Alpha-Beta (AB) and
Gamma Log-det divergences many well known divergences, for example, the Stein's
loss, S-divergence, called also Jensen-Bregman LogDet (JBLD) divergence, the
Logdet Zero (Bhattacharryya) divergence, Affine Invariant Riemannian Metric
(AIRM) as well as some new divergences. Moreover, we establish links and
correspondences among many log-det divergences and display them on alpha-beta
plain for various set of parameters. Furthermore, this paper bridges these
divergences and shows also their links to divergences of multivariate and
multiway Gaussian distributions. Closed form formulas are derived for gamma
divergences of two multivariate Gaussian densities including as special cases
the Kullback-Leibler, Bhattacharryya, R\'enyi and Cauchy-Schwartz divergences.
Symmetrized versions of the log-det divergences are also discussed and
reviewed. A class of divergences is extended to multiway divergences for
separable covariance (precision) matrices.
Reciprocity is a vital feature of social networks, but little is known about
its structure in real world behavior, or the mechanisms underlying its
persistence. In pursuit of these two questions, we study the stationary and
dynamical signals of reciprocity in a network of manioc beer drinking events in
a Tsimane' village in lowland Bolivia. At the stationary level, our analysis
reveals that social exchange within the community is heterogenous, even when
controlling for kinship. A positive relationship between the frequencies at
which two families host each other provides evidence for stationary
reciprocity. Our analysis of the dynamical structure of this network presents a
novel method for the study of conditional, or non-stationary, reciprocity
effects. We find that levels of cooperation seen over the sixteen-week study
period can be accounted for on the stationary hypothesis alone. This suggests
that `tit for tat' effects do not play an observable role in maintaining the
high levels of stationary reciprocation seen in this system.
Since Renes et al. [J. Math. Phys. 45, 2171 (2004)], there has been much effort in the quantum information community to prove (or disprove) the existence of symmetric informationally complete (SIC) sets of quantum states in arbitrary finite dimension. This paper strengthens the urgency of this question by showing that if SIC-sets exist: 1) by a natural measure of orthonormality, they are as close to being an orthonormal basis for the space of density operators as possible, and 2) in prime dimensions, the standard construction for complete sets of mutually unbiased bases and Weyl-Heisenberg covariant SIC-sets are intimately related: The latter represent minimum uncertainty states for the former in the sense of Wootters and Sussman. Finally, we contribute to the question of existence by conjecturing a quadratic redundancy in the equations for Weyl-Heisenberg SIC-sets. Comment: 4 pages, no figures, revtex. Version 2: new title, result in Section 1 strengthened, otherwise minor changes only
A stochastic nonlinear dynamical system generates information, as measured by
its entropy rate. Some---the ephemeral information---is dissipated and
some---the bound information---is actively stored and so affects future
behavior. We derive analytic expressions for the ephemeral and bound
informations in the limit of small-time discretization for two classical
systems that exhibit dynamical equilibria: first-order Langevin equations (i)
where the drift is the gradient of a potential function and the diffusion
matrix is invertible and (ii) with a linear drift term (Ornstein-Uhlenbeck) but
a noninvertible diffusion matrix. In both cases, the bound information is
sensitive only to the drift, while the ephemeral information is sensitive only
to the diffusion matrix and not to the drift. Notably, this information anatomy
changes discontinuously as any of the diffusion coefficients vanishes,
indicating that it is very sensitive to the noise structure. We then calculate
the information anatomy of the stochastic cusp catastrophe and of particles
diffusing in a heat bath in the overdamped limit, both examples of stochastic
gradient descent on a potential landscape. Finally, we use our methods to
calculate and compare approximations for the so-called time-local predictive
information for adaptive agents.
A directed acyclic graph (DAG) partially represents the conditional independence structure among observations of a system if the local Markov condition holds, that is, if every variable is independent of its non-descendants given its parents. In general, there is a whole class of DAGs that represents a given set of conditional independence relations. We are interested in properties of this class that can be derived from observations of a subsystem only. To this end, we prove an information theoretic inequality that allows for the inference of common ancestors of observed parts in any DAG representing some unknown larger system. More explicitly, we show that a large amount of dependence in terms of mutual information among the observations implies the existence of a common ancestor that distributes this information. Within the causal interpretation of DAGs our result can be seen as a quantitative extension of Reichenbach's Principle of Common Cause to more than two variables. Our conclusions are valid also for non-probabilistic observations such as binary strings, since we state the proof for an axiomatized notion of mutual information that includes the stochastic as well as the algorithmic version. Comment: 18 pages, 4 figures
In this paper, new techniques that allow conditional entropy to estimate the
combinatorics of symbols are applied to animal communication studies relying on
information theory. By using the conditional entropy estimates at multiple
orders, the paper estimates the total repertoire sizes for animal communication
across bottlenose dolphins, humpback whales, and several species of birds for
N-grams of one, two, and three combined units. How this can influence our
estimates and ideas about the complexity of animal communication is also
discussed.
Previous studies modelled the origin of life and the emergence of photosynthesis on the early Earth-i.e. the origin of plants-in terms of biological heat engines that worked on thermal cycling caused by suspension in convecting water. In this new series of studies, heat engines using a more complex mechanism for thermal cycling are invoked to explain the origin of animals as well. Biological exploitation of the thermal gradient above a submarine hydrothermal vent is hypothesized, where a relaxation oscillation in the length of a protein 'thermotether' would have yielded the thermal cycling required for thermosynthesis. Such a thermal transition driven movement is not impeded by the low Reynolds number of a small scale. In the model the thermotether together with the protein export apparatus evolved into a 'flagellar proton pump' that turned into today's bacterial flagellar motor after the acquisition of the proton-pumping respiratory chain. The flagellar pump resembles Feynman's ratchet, and the 'flagellar computer' that implements chemotaxis a Turing machine: the stator would have functioned as Turing's paper tape and the stator's proton-transferring subunits with their variable conformation as the symbols on the tape. The existence of a cellular control centre in the cilium of the eukaryotic cell is proposed that would have evolved from the prokaryotic flagellar computer. Comment: Correction of text on Turing machines; Appendix added with journal reviewer's criticism, and reaction to that criticism; minor typos
Mutual information among three or more dimensions (mu-star = - Q) has been considered as interaction information. However, Krippendorff (2009a, 2009b) has shown that this measure cannot be interpreted as a unique property of the interactions and has proposed an alternative measure of interaction information based on iterative approximation of maximum entropies. Q can then be considered as a measure of the difference between interaction information and redundancy generated in a model entertained by an observer. I argue that this provides us with a measure of the imprint of a second-order observing system -- a model entertained by the system itself -- on the underlying information processing. The second-order system communicates meaning hyper-incursively; an observation instantiates this meaning-processing within the information processing. The net results may add to or reduce the prevailing uncertainty. The model is tested empirically for the case where textual organization can be expected to contain intellectual organization in terms of distributions of title words, author names, and cited references.
Conventional thermodynamics, which is formulated for our world populated by
radiation and matter, can be extended to describe physical properties of
antimatter in two mutually exclusive ways: CP-invariant or CPT-invariant. Here
we refer to invariance of physical laws under charge (C), parity (P) and time
reversal (T) transformations. While in quantum field theory CPT invariance is a
theorem confirmed by experiments, the symmetry principles applied to
macroscopic phenomena or to the whole of the Universe represent only
hypotheses. Since both versions of thermodynamics are different only in their
treatment of antimatter, but are the same in describing our world dominated by
matter, making a clear experimentally justified choice between CP invariance
and CPT invariance in context of thermodynamics is not possible at present.
This work investigates the comparative properties of the CP- and CPT-invariant
extensions of thermodynamics (focusing on the latter, which is less
conventional than the former) and examines conditions under which these
extensions can be experimentally tested.
Stirling approximation of the factorials and multinominal coefficients are generalized based on the one-parameter ($\kappa$) deformed functions introduced by Kaniadakis [Phys. Rev. E \textbf{66} (2002) 056125]. We have obtained the relation between the $\kappa$-generalized multinominal coefficients and the $\kappa$-entropy by introducing a new $\kappa$-product operation.
Herein, we present a feasible, general protocol for quantum communication
within a network via generalized remote preparation of an arbitrary $m$-qubit
entangled state designed with genuine tripartite
Greenberger--Horne--Zeilinger-type entangled resources. During the
implementations, we construct novel collective unitary operations; these
operations are tasked with performing the necessary phase transfers during
remote state preparations. We have distilled our implementation methods into a
five-step procedure, which can be used to faithfully recover the desired state
during transfer. Compared to previous existing schemes, our methodology
features a greatly increased success probability. After the consumption of
auxiliary qubits and the performance of collective unitary operations, the
probability of successful state transfer is increased four-fold and eight-fold
for arbitrary two- and three-qubit entanglements when compared to other methods
within the literature, respectively. We conclude this paper with a discussion
of the presented scheme for state preparation, including: success
probabilities, reducibility and generalizability.
Renewal processes are broadly used to model stochastic behavior consisting of
isolated events separated by periods of quiescence, whose durations are
specified by a given probability law. Here, we identify the minimal sufficient
statistic for their prediction (the set of causal states), calculate the
historical memory capacity required to store those states (statistical
complexity), delineate what information is predictable (excess entropy), and
decompose the entropy of a single measurement into that shared with the past,
future, or both. The causal state equivalence relation defines a new subclass
of renewal processes with a finite number of causal states despite having an
unbounded interevent count distribution. We apply our new formulae for
information measures to analyze the output of the parametrized simple
nonunifilar source, a simple two-state machine with an infinite-state
epsilon-machine presentation. All in all, the results lay the groundwork for
analyzing processes with divergent statistical complexity and divergent excess
entropy.
The recent development of the theory of fluctuation relations has led to new
insights into the ever-lasting question of how irreversible behavior emerges
from time-reversal symmetric microscopic dynamics. We provide an introduction
to fluctuation relations, examine their relation to dissipation and discuss
their impact on the arrow of time question.
I argue that opposite arrows of time, while being logically possible, cannot realistically be assumed to exist during one and the same epoch of this universe.
Finding interdependency relations between (possibly multivariate) time series provides valuable knowledge about the processes that generate the signals. Information theory sets a natural framework for non-parametric measures of several classes of statistical dependencies. However, a reliable estimation from information-theoretic functionals is hampered when the dependency to be assessed is brief or evolves in time. Here, we show that these limitations can be overcome when we have access to an ensemble of independent repetitions of the time series. In particular, we gear a data-efficient estimator of probability densities to make use of the full structure of trial-based measures. By doing so, we can obtain time-resolved estimates for a family of entropy combinations (including mutual information, transfer entropy, and their conditional counterparts) which are more accurate than the simple average of individual estimates over trials. We show with simulated and real data that the proposed approach allows to recover the time-resolved dynamics of the coupling between different subsystems.
We study a system represented by a Bose-Einstein condensate interacting with
a cavity field in presence of a strong off-resonant pumping laser. This system
can be described by a three-mode Gaussian state, where two are the atomic modes
corresponding to atoms populating upper and lower momentum sidebands and the
third mode describes the scattered cavity field light. We show that, as a
consequence of the collective atomic recoil instability, these modes possess a
genuine tripartite entanglement that increases unboundedly with the evolution
time and is larger than the bipartite entanglement in any reduced two-mode
bipartition. We further show that the state of the system exhibits genuine
tripartite nonlocality, which can be revealed by a robust violation of the
Svetlichny inequality when performing displaced parity measurements. Our exact
results are obtained by exploiting the powerful machinery of phase-space
informational measures for Gaussian states, which we briefly review in the
opening sections of the paper.
The paper presents an approach to olfactory search for a diffusive emitting
source of tracer (e.g. aerosol, gas) in an environment with unknown map of
randomly placed and shaped obstacles.
The measurements of tracer concentration are sporadic, noisy and without
directional information. The search domain is discretised and modelled by a
finite two-dimensional lattice. The links is the lattice represent the
traversable paths for emitted particles and for the searcher. A missing link in
the lattice indicates a blocked paths, due to the walls or obstacles. The
searcher must simultaneously estimate the source parameters, the map of the
search domain and its own location within the map. The solution is formulated
in the sequential Bayesian framework and implemented as a Rao-Blackwellised
particle filter with information-driven motion control. The numerical results
demonstrate the concept and its performance.
Over the last decade it has been found that nonlinear laws of composition of
momenta are predicted by some alternative approaches to "real" 4D quantum
gravity, and by all formulations of dimensionally-reduced (3D) quantum gravity
coupled to matter. The possible relevance for rather different quantum-gravity
models has motivated several studies, but this interest is being tempered by
concerns that a nonlinear law of addition of momenta might inevitably produce a
pathological description of the total momentum of a macroscopic body. I here
show that such concerns are unjustified, finding that they are rooted in
failure to appreciate the differences between two roles for laws composition of
momentum in physics. Previous results relied exclusively on the role of a law
of momentum composition in the description of spacetime locality. However, the
notion of total momentum of a multi-particle system is not a manifestation of
locality, but rather reflects translational invariance. By working within an
illustrative example of quantum spacetime I show explicitly that spacetime
locality is indeed reflected in a nonlinear law of composition of momenta, but
translational invariance still results in an undeformed linear law of addition
of momenta building up the total momentum of a multi-particle system.
We consider questions posed in a recent paper of Mandayam, Bandyopadhyay,
Grassl and Wootters [10] on the nature of "unextendible mutually unbiased
bases." We describe a conceptual framework to study these questions, using a
connection proved by the author in [19] between the set of nonidentity
generalized Pauli operators on the Hilbert space of $N$ $d$-level quantum
systems, $d$ a prime, and the geometry of non-degenerate alternating bilinear
forms of rank $N$ over finite fields $\mathbb{F}_d$. We then supply alternative
and short proofs of results obtained in [10], as well as new general bounds for
the problems considered in loc. cit. In this setting, we also solve Conjecture
1 of [10], and speculate on variations of this conjecture.
Real-world social and economic networks typically display a number of
particular topological properties, such as a giant connected component, a broad
degree distribution, the small-world property and the presence of communities
of densely interconnected nodes. Several models, including ensembles of
networks also known in social science as Exponential Random Graphs, have been
proposed with the aim of reproducing each of these properties in isolation.
Here we define a generalized ensemble of graphs by introducing the concept of
graph temperature, controlling the degree of topological optimization of a
network. We consider the temperature-dependent version of both existing and
novel models and show that all the aforementioned topological properties can be
simultaneously understood as the natural outcomes of an optimized,
low-temperature topology. We also show that seemingly different graph models,
as well as techniques used to extract information from real networks, are all
found to be particular low-temperature cases of the same generalized formalism.
One such technique allows us to extend our approach to real weighted networks.
Our results suggest that a low graph temperature might be an ubiquitous
property of real socio-economic networks, placing conditions on the diffusion
of information across these systems.
Hawking radiation and Bekenstein--Hawking entropy are the two robust
predictions of a yet unknown quantum theory of gravity. Any theory which fails
to reproduce these predictions is certainly incorrect. While several approaches
lead to Bekenstein--Hawking entropy, they all lead to different sub-leading
corrections. In this article, we ask a question that is relevant for any
approach: Using simple techniques, can we know whether an approach contains
quantum or semi-classical degrees of freedom? Using naive dimensional analysis,
we show that the semi-classical black-hole entropy has the same dimensional
dependence as the gravity action. Among others, this provides a plausible
explanation for the connection between Einstein's equations and thermodynamic
equation of state, and that the quantum corrections should have a different
scaling behavior.
We challenge claims that the principle of maximum entropy production produces
physical phenomenological relations between conjugate currents and forces, even
beyond the linear regime, and that currents in networks arrange themselves to
maximize entropy production as the system approaches the steady state. In
particular: (1) we show that Ziegler's principle of thermodynamic orthogonality
leads to stringent reciprocal relations for higher order response coe?cients,
and in the framework of stochastic thermodynamics, we exhibit a simple explicit
model that does not satisfy them; (2) on a network, enforcing Kirchhoff's
current law, we show that maximization of the entropy production prescribes
reciprocal relations between coarse-grained observables, but is not responsible
for the onset of the steady state, which is rather due to the minimum entropy
production principle.
In thermodynamics one considers thermal systems and the maximization of
entropy subject to the conservation of energy. A consequence is Landauer's
erasure principle, which states that the erasure of 1 bit of information
requires a minimum energy cost equal to $kT\ln(2)$ where $T$ is the temperature
of a thermal reservoir used in the process and $k$ is Boltzmann's constant.
Jaynes, however, argued that the maximum entropy principle could be applied to
any number of conserved quantities which would suggest that information erasure
may have alternative costs. Indeed we showed recently that by using a reservoir
comprising energy degenerate spins and subject to conservation of angular
momentum, the cost of information erasure is in terms of angular momentum
rather than energy. Here we extend this analysis and derive the minimum cost of
information erasure for systems where different conservation laws operate. We
find that, for each conserved quantity, the minimum resource needed to erase 1
bit of memory is $\lambda^{-1}\ln(2)$ where $\lambda$ is related to the average
value of the conserved quantity. The costs of erasure depend, fundamentally, on
both the nature of the physical memory element and the reservoir with which it
is coupled.
Nanothermodynamics extends standard thermodynamics to facilitate finite-size
effects on the scale of nanometers. A key ingredient is Hill's subdivision
potential that accommodates the non-extensive energy of independent small
systems, similar to how Gibbs' chemical potential accommodates distinct
particles. Nanothermodynamics is essential for characterizing the thermal
equilibrium distribution of independently relaxing regions inside bulk samples,
as is found for the primary response of most materials using various
experimental techniques. The subdivision potential ensures strict adherence to
the laws of thermodynamics: total energy is conserved by including an
instantaneous contribution from the entropy of local configurations, and total
entropy remains maximized by coupling to a thermal bath. A unique feature of
nanothermodynamics is the completely-open nanocanonical ensemble. Another
feature is that particles within each region become statistically
indistinguishable, which avoids non-extensive entropy, and mimics
quantum-mechanical behavior. Applied to mean-field theory, nanothermodynamics
gives a heterogeneous distribution of regions that yields stretched-exponential
relaxation and super-Arrhenius activation. Applied to Monte Carlo simulations,
there is a nonlinear correction to Boltzmann's factor that improves agreement
between the Ising model and measured non-classical critical scaling in magnetic
materials. Nanothermodynamics also provides a fundamental mechanism for the 1/f
noise found in many materials.
The existing upper and lower bounds between entropy and error are mostly
derived through an inequality means without linking to joint distributions. In
fact, from either theoretical or application viewpoint, there exists a need to
achieve a complete set of interpretations to the bounds in relation to joint
distributions. For this reason, in this work we propose a new approach of
deriving the bounds between entropy and error from a joint distribution. The
specific case study is given on binary classifications, which can justify the
need of the proposed approach. Two basic types of classification errors are
investigated, namely, the Bayesian and non-Bayesian errors. For both errors, we
derive the closed-form expressions of upper bound and lower bound in relation
to joint distributions. The solutions show that Fano's lower bound is an exact
bound for any type of errors in a relation diagram of "Error Probability vs.
Conditional Entropy". A new upper bound for the Bayesian error is derived with
respect to the minimum prior probability, which is generally tighter than
Kovalevskij's upper bound.
Maximum entropy models are increasingly being used to describe the collective activity of neural populations with measured mean neural activities and pairwise correlations, but the full space of probability distributions consistent with these constraints has not been explored. We provide upper and lower bounds on the entropy for the {\em minimum} entropy distribution over arbitrarily large collections of binary units with any fixed set of mean values and pairwise correlations. We also construct specific low-entropy distributions for several relevant cases. Surprisingly, the minimum entropy solution has entropy scaling logarithmically with system size for any set of first- and second-order statistics consistent with arbitrarily large systems. We further demonstrate that some sets of these low-order statistics can only be realized by small systems. Our results show how only small amounts of randomness are needed to mimic low-order statistical properties of highly entropic distributions, and we discuss some applications for engineered and biological information transmission systems.
Free energy and entropy are examined in detail from the standpoint of classical thermodynamics. The approach is logically based on the fact that thermodynamic work is mediated by thermal energy through the tendency for nonthermal energy to convert spontaneously into thermal energy and for thermal energy to distribute spontaneously and uniformly within the accessible space. The fact that free energy is a Second-Law, expendable energy that makes it possible for thermodynamic work to be done at finite rates is emphasized. Entropy, as originally defined, is pointed out to be the capacity factor for thermal energy that is hidden with respect to temperature; it serves to evaluate the practical quality of thermal energy and to account for changes in the amounts of latent thermal energies in systems maintained at constant temperature. A major objective was to clarify the means by which free energy is transferred and conserved in sequences of biological reactions coupled by freely diffusible intermediates. In achieving this objective it was found necessary to distinguish between a 'characteristic free energy' possessed by all First-Law energies in amounts equivalent to the amounts of the energies themselves and a 'free energy of concentration' that is intrinsically mechanical and relatively elusive in that it can appear to be free of First-Law energy. The findings in this regard serve to clarify the fact that the transfer of chemical potential energy from one repository to another along sequences of biological reactions of the above sort occurs through transfer of the First-Law energy as thermal energy and transfer of the Second-Law energy as free energy of concentration. Comment: 18-page PDF; major correction in APPENDIX; minor corrections elsewhere
We developed an automated pattern recognition code that is particularly well
suited to extract one-dimensional curvi-linear features from two-dimensional
digital images. A former version of this {\sl Oriented Coronal CUrved Loop
Tracing (OCCULT)} code was applied to spacecraft images of magnetic loops in
the solar corona, recorded with the NASA spacecraft {\sl Transition Region And
Coronal Explorer (TRACE)} in extreme ultra-violet wavelengths. Here we apply an
advanced version of this code ({\sl OCCULT-2}) also to similar images from the
{\sl Solar Dynamics Observatory (SDO)}, to chromospheric H-$\alpha$ images
obtained with the {\sl Swedish Solar Telescope (SST)}, and to microscopy images
of microtubule filaments in live cells in biophysics. We provide a full
analytical description of the code, optimize the control parameters, and
compare the automated tracing with visual/manual methods. The traced structures
differ by up to 16 orders of magnitude in size, which demonstrates the
universality of the tracing algorithm.
Information geometric optimization (IGO) is a general framework for
stochastic optimization problems aiming at limiting the influence of arbitrary
parametrization choices. The initial problem is transformed into the
optimization of a smooth function on a Riemannian manifold, defining a
parametrization-invariant first order differential equation. However, in
practice, it is necessary to discretize time, and then, parametrization
invariance holds only at first order in the step size.
We define the Geodesic IGO update (GIGO), which uses the Riemannian manifold
structure to obtain an update entirely independent from the parametrization of
the manifold. We test it with classical objective functions.
Thanks to Noether's theorem from classical mechanics, we find a reasonable
way to compute the geodesics of the statistical manifold of Gaussian
distributions, and thus the corresponding GIGO update. We then compare GIGO,
pure rank-$\mu$ CMA-ES and xNES (two previous algorithms that can be recovered
by the IGO framework), and show that while the GIGO and xNES updates coincide
when the mean is fixed, they are different in general, contrary to previous
intuition. We then define a new algorithm (Blockwise GIGO) that recovers the
xNES update from abstract principles.
From the perspective of the statistical fluctuation theory, we explore the
role of the thermodynamic geometries and vacuum (in)stability properties for
the topological Einstein-Yang-Mills black holes. In this paper, from the
perspective of the state-space surface and chemical Wienhold surface, we
provide the criteria for the local and global statistical stability of an
ensemble of topological Einstein-Yang-Mills black holes in arbitrary spacetime
dimensions $D\ge 5$. Finally, as per the formulations of the thermodynamic
geometry, we offer a parametric account of the statistical consequences in both
the local and global fluctuation regimes of the topological Einstein-Yang-Mills
black holes.