The generalization performance of ERM algorithm with strongly mixing observations

Abstract

The generalization performance is the main concern of machine learning theoretical research. The previous main bounds describing
the generalization ability of the Empirical Risk Minimization (ERM) algorithm are based on independent and identically distributed
(i.i.d.) samples. In order to study the generalization performance of the ERM algorithm with dependent observations, we first
establish the exponential bound on the rate of relative uniform convergence of the ERM algorithm with exponentially strongly
mixing observations, and then we obtain the generalization bounds and prove that the ERM algorithm with exponentially strongly
mixing observations is consistent. The main results obtained in this paper not only extend the previously known results for
i.i.d. observations to the case of exponentially strongly mixing observations, but also improve the previous results for strongly
mixing samples. Because the ERM algorithm is usually very time-consuming and overfitting may happen when the complexity of
the hypothesis space is high, as an application of our main results we also explore a new strategy to implement the ERM algorithm
in high complexity hypothesis space.