[show abstract][hide abstract] ABSTRACT: Models of dynamical systems based on predictive state representations (PSRs)
are defined strictly in terms of observable quantities, in contrast with
traditional models (such as Hidden Markov Models) that use latent variables or
statespace representations. In addition, PSRs have an effectively infinite
memory, allowing them to model some systems that finite memory-based models
cannot. Thus far, PSR models have primarily been developed for domains with
discrete observations. Here, we develop the Predictive Linear-Gaussian (PLG)
model, a class of PSR models for domains with continuous observations. We show
that PLG models subsume Linear Dynamical System models (also called Kalman
filter models or state-space models) while using fewer parameters. We also
introduce an algorithm to estimate PLG parameters from data, and contrast it
with standard Expectation Maximization (EM) algorithms used to estimate Kalman
filter parameters. We show that our algorithm is a consistent estimation
procedure and present preliminary empirical results suggesting that our
algorithm outperforms EM, particularly as the model dimension increases.
[show abstract][hide abstract] ABSTRACT: Most work on Predictive Representations of State (PSRs) has focused on learning and planning in un- structured domains (for example, those represented by flat POMDPs). This paper extends PSRs to rep- resent relational knowledge about domains, so that they can use policies that generalize across differ- ent tasks, capture knowledge that ignores irrele- vant attributes of objects, and represent policies in a way that is independent of the size of the state space. Using a blocks world domain, we show how generalized predictions about the future can com- pactly capture relations between objects, which in turn can be used to naturally specify relational-style options and policies. Because our representation is expressed solely in terms of actions and observa- tions, it has extensive semantics which are statistics about observable quantities.
[show abstract][hide abstract] ABSTRACT: Models of agent-environment interaction that use predic- tive state representations (PSRs) have mainly focused on the case of discrete observations and actions. The theory of discrete PSRs uses an elegant construct called the system dynamics matrix and derives the notion of predictive state as a sufficient statistic via the rank of the matrix. With continuous observations and actions, such a matrix and its rank no longer exist. In this paper, we show how to define an analogous construct for the continuous case, called the sys- tem dynamics distributions, and use information theoretic notions to define a sufficient statistic and thus state. Given this new construct, we use kernel density estimation to learn approximate system dynamics distributions from data, and use information-theoretic tools to derive algorithms for dis- covery of state and learning of model parameters. We illus- trate our new modeling method on two example problems.