This study employs a Luenberger-Hicks-Moorsteen productivity indicator to measure the total factor energy efficiency (TFEE) of the Yangtze River Economic Belt (YREB) of China from 2005 to 2016. The relative importance analysis method is used to investigate the driving forces of the TFEE. The main results are as follows. (a) The TFEE has an average growth rate of 3.3% during the study period. The TFEE declines first, then rebounds, and declines again. The contribution of technological efficiency changes to the TFEE growth is almost zero which indicating no catch-up effect in the sample period. Technological progress contributes a negative impact to the TFEE, with an average annual decrease rate of 3.4%. The scale efficiency changes, with an average annual growth rate of 6.7%, are extremely important for the growth of TFEE. (b) The TFEE of the western region is narrowing the gap with the central and western regions, while the gap between the central region and the eastern region is increasing. (c) Research investment can improve the growth of TFEE significantly, while government expenditure and industrial structure are not. And government expenditure, economic development, and research investment are top factors to explain the variation of TFEE.