Discriminative Sparsification and Binarization of Convolution Neural Networks
Abstract
Convolution neural networks have become one of the dominating deep learning models, especially for computation vision tasks such as image classification and segmentation. Dense convolution filters are inefficient, due to huge amount of full precision multiplications involved in their computation. Motivated by previous successes on sparse models and binary models, we propose a Max-Min technique to train sparse and binary convolution neural networks with fixed structures. As opposed to previous methods, the network structure is consistent during the training phase, which has potential advantage with respect to power and memory management for hardware implementation. Computation complexity of the proposed techniques is analyzed to compare with the dense structures, showing significant reduction on the number of floating point operations (FLOPs). Specific techniques, including training tricks and structural augmentation, are discussed to facilitate fast and correct convergence, and to alleviate potential accuracy degradation introduced by these accelerating techniques. The proposed techniques are applied on several most successful architectures to obtain their sparse and binary versions, including AlexNet for both techniques, VGGNet, Inception-v3, and ResNet for sparsification. Extensive experiments on benchmark datasets (MNIST and CIFAR10) are conducted, demonstrating the effectiveness of the proposed techniques empirically with comparable prediction performance between the original dense models and their sparse or binary versions.
Citation
Jin, Qing (2018). Discriminative Sparsification and Binarization of Convolution Neural Networks. Master's thesis, Texas A&M University. Available electronically from https : / /hdl .handle .net /1969 .1 /192040.