|Flow Of Renyi Information In Deep Neural Networks|
|Che-Wei Huang, SAIL, University of Southern California|
Shrikanth S. Narayanan, SAIL, University of Southern California
We propose a rate-distortion based deep neural network (DNN) training algorithm using a smooth matrix functional on the manifold of positive semi-definite matrices as the non-parametric entropy estimator. The objective in the op- timization function includes not only the measure of perfor- mance of the output layer but also the measure of information distortion between consecutive layers in order to produce a concise representation of its input on each layer. An experi- ment on speech emotion recognition shows the DNN trained by such method reaches comparable performance with an encoder-decoder system.