Self-normalizing Neural Networks

Goal: Self-normalizing Neural Networks automatically perform normalization like batch normalization or layer normalization via moving towards a stable fixed point. This stable fixed point allows optimal learning without vanishing or exploding gradients. This stable fixed point is moment conserving (mean activation is tends towards zero and the variance tends towards one). Self-normalizing Neural Networks are the key to construct large AI systems since building blocks can be combined while these blocks ensure efficient learning via self-normalization.