Flow Of Renyi Information In Deep Neural Networks
Che-Wei Huang, SAIL, University of Southern California
Shrikanth S. Narayanan, SAIL, University of Southern California

Abstract:
We propose a rate-distortion based deep neural network (DNN) training algorithm using a smooth matrix functional on the manifold of positive semi-definite matrices as the non-parametric entropy estimator. The objective in the op- timization function includes not only the measure of perfor- mance of the output layer but also the measure of information distortion between consecutive layers in order to produce a concise representation of its input on each layer. An experi- ment on speech emotion recognition shows the DNN trained by such method reaches comparable performance with an encoder-decoder system.