Can We Gain More from Orthogonality Regularizations in Training Deep Networks?
Abstract
1This work seeks to answer the question: as the (near-) orthogonality of weights is found to
be a favorable property for training deep convolutional neural networks, how we can enforce
it in more effective and easy-to-use ways? Through this work we look to come up with novel
orthogonality regularizations for training deep CNNs, utilizing various advanced analytical tools
such as mutual coherence and Restricted Isometric Property. These plug-and-play regularizations
can be conveniently incorporated into training almost any CNN without extra hassle. We then
benchmark their effects on three state-of-the-art models: ResNet, WideResNet, and ResNeXt,
on CIFAR-10 and CIFAR-100 and SVHN datasets. To validate method’s efficacy across various
distribution and dataset, we apply the best performing regularizer(SRIP), for different setting of
WideResNet to ImageNet Dataset. We observe consistent performance gains after applying those
proposed regularizations, in terms of both the final accuracies achieved, and accelerated and more
stable convergences.
1Reprinted with permission from Abstract section of Can We Gain More from Orthogonality Regularizations in
Training Deep Networks? by N. Bansal, X. Chen and Z. Wang, 2018,Advances in Neural Information Processing
Systems 31 (NIPS 2018) pre-proceedings
Citation
Bansal, Nitin Kumar (2018). Can We Gain More from Orthogonality Regularizations in Training Deep Networks?. Master's thesis, Texas A & M University. Available electronically from https : / /hdl .handle .net /1969 .1 /174565.