Dynamic Analysis of Recurrent Neural Networks
Abstract
With the advancement in deep learning research, neural networks have become one of the most powerful tools for artificial intelligence tasks. More specifically, recurrent neural networks (RNNs) have achieved state-of-the-art in tasks such as hand-writing recognition and speech recognition. Despite the success of recurrent neural networks, how and why do neural nets work is still not sufficiently investigated. My work on the dynamical analysis of recurrent neural networks can help understand how the input features are extracted in the recurrent layer, how the RNNs make decisions, and how the chaotic dynamics of RNNs affects its behaviors. Firstly, I investigated the dynamics of recurrent neural networks as autonomous dynamical system in the experiment of a two-joint limb controlling task and compared the empirical result and the theoretical analysis. Secondly, I investigated the dynamics of non-autonomous recurrent neural networks on two benchmark tasks: sequential MNIST recognition task and DNA splice junction classification task. How the hidden states of long-short term memory (LSTM) and gated recurrent unit (GRU) cells learn new features and how the input sequence is extracted are demonstrated with experiments. Finally, based on the understanding of the external and internal dynamics of recurrent units, I proposed several algorithms for recurrent neural network compression. The algorithms demonstrate reasonable performance in compression ratio and are able to sustain the performance of the original models.
Citation
Wang, Han (2020). Dynamic Analysis of Recurrent Neural Networks. Doctoral dissertation, Texas A&M University. Available electronically from https : / /hdl .handle .net /1969 .1 /191907.