My work (Learning Intrinsic Sparse Structures within Long Short-Term Memory) on structurally sparse Recurrent Neural Networks (RNNs) is accepted by ICLR 2018. This is a work collaborated with Microsoft Research and it’s advanced from my NIPS 2016 paper on structurally sparse Convolutional Neural Networks (CNNs). Our source code is available on my GitHub. Poster is here.

By learning structurally sparse RNNs, we can reduce the hidden size in RNNs so as to accelerate inference and save memory. The regularity of the sparse weights enables a higher utilization of the peak computing capacity in AI platforms, and gives higher speedup than random connection pruning as shown in the top figure. A later OpenAI work of “Block-Sparse GPU Kernels” proves both the training and inference efficiency of structurally sparse RNNs.

See you in Vancouver!!!