Towards Robust Design and Training of Deep Neural Networks
Abstract
Currently neural networks run as software, which typically requires expensive GPU resources. As the adoption of deep learning continues for a more diverse range of applications, direct hardware implemented neural networks (HNN) will provide deep learning solutions at far lower hardware requirements. However, Gaussian noise along hardware connections degrades model accuracy, an issue this research seeks to resolve using a novel analog error correcting code (ECC). To aid in developing noise tolerant deep neural networks (DNN), this research also investigates the impact of loss functions on training. This involves alternating multiple loss functions throughout training, aiming to prevent local optimals. The effects on training time and final accuracy are then analyzed. This research investigates analog ECCs and loss function variation to allow for future noise tolerant HNN networks. ECC results demonstrate three to five decibel improvements to model accuracy when correcting Gaussian noise. Loss variation results demonstrate a correlation between loss function similarity and training performance. Other correlations are also presented and addressed.
Subject
deep learningneural networks
robust neural networks
robust training
gaussian noise
hardware neural networks
error correcting codes
analog error correcting codes
loss variation
loss functions
global optimals
facial expression recognition
computer hardware
Citation
Cordero, Jeffrey Dean (2019). Towards Robust Design and Training of Deep Neural Networks. Undergraduate Research Scholars Program. Available electronically from https : / /hdl .handle .net /1969 .1 /175414.