We establish upper bounds for the minimal number of hidden units for which a
binary stochastic feedforward network with sigmoid activation probabilities and
a single hidden layer is a universal approximator of Markov kernels. We show
that each possible probabilistic assignment of the states of $n$ output units,
given the states of $k\geq1$ input units, can be approximated arbitrarily well
by a... [Show full abstract]
We show that deep narrow Boltzmann machines are universal approximators of probability distributions on the activities of their visible units, provided they have sufficiently many hidden layers, each containing the same number of units as the visible layer. We show that, within certain parameter domains, deep Boltzmann machines can be studied as feedforward networks. We provide upper and lower... [Show full abstract]
We generalize recent theoretical work on the minimal number of layers of narrow deep belief networks that can approximate any probability distribution on the states of their visible units arbitrarily well, from the setting of binary units (Sutskever and Hinton, 2008; Le Roux and Bengio, 2008, 2010; Montúfar and Ay, 2011) to the setting of units with finite state spaces. In particular we show... [Show full abstract]
We follow up on previous work addressing the number of response regions of the functions representable by feedforward neural networks with piecewise linear activation functions. We discuss upper bounds on the maximum number of linear regions for deep networks with rectified linear units. We elaborate on the identification of input regions as an analysis tool, and how it implies exponential... [Show full abstract]
Data provided are for informational purposes only. Although carefully collected, accuracy cannot be guaranteed. Publisher conditions are provided by RoMEO. Differing provisions from the publisher's actual policy or licence agreement may be applicable.
This publication is from a journal that may support self archiving.