Can We Gain More from Orthogonality Regularizations in Training Deep Networks?

Abstract

1This work seeks to answer the question: as the (near-) orthogonality of weights is found to be a favorable property for training deep convolutional neural networks, how we can enforce it in more effective and easy-to-use ways? Through this work we look to come up with novel orthogonality regularizations for training deep CNNs, utilizing various advanced analytical tools such as mutual coherence and Restricted Isometric Property. These plug-and-play regularizations can be conveniently incorporated into training almost any CNN without extra hassle. We then benchmark their effects on three state-of-the-art models: ResNet, WideResNet, and ResNeXt, on CIFAR-10 and CIFAR-100 and SVHN datasets. To validate method’s efficacy across various distribution and dataset, we apply the best performing regularizer(SRIP), for different setting of WideResNet to ImageNet Dataset. We observe consistent performance gains after applying those proposed regularizations, in terms of both the final accuracies achieved, and accelerated and more stable convergences. 1Reprinted with permission from Abstract section of Can We Gain More from Orthogonality Regularizations in Training Deep Networks? by N. Bansal, X. Chen and Z. Wang, 2018,Advances in Neural Information Processing Systems 31 (NIPS 2018) pre-proceedings