Hardware Testbed and Deep Neural Networks for Multi-Modal Sensor Fusion

Abstract

Deep neural networks (DNN) have been widely applied in sensor fusion, providing an end-to-end solution for fusion of features extracted from multiple sensory inputs. A class of new sensor fusion networks based on DNN called gating architectures proposed in recent years improves the prediction performances over the conventional fusion mechanisms employed in convolutional neural networks (CNNs). However, experimental results show that the gating architectures are not always robust and sometimes even underperform conventional fusion methods. In this work, the limitations of existing gating architectures are discussed and analyzed. Through experiments, we demonstrate that gating architectures fail to learn correct fusion weights for sensory inputs, showing the inconsistency between fusion weights and corresponding qualities of sensory inputs, and hence limit the prediction performance. We propose an improved fusion architecture by introducing the auxiliary path model to regulate the fusion weights in the gating architecture. We also provide in-depth studies on the regularization mechanisms to show that the improvements on performances are achieved by the more robustly learnt fusion weights. Evaluations are performed under two different public datasets. We generate comprehensive sensor failure schemes, where the proposed architecture significantly outperforms a baseline non-gating architecture and one existing gating architecture. We also build up a sensor fusion hardware platform: a robot car, which is equipped with multiple sensors. The robot will be further developed and adopted as a hardware platform for evaluating the proposed sensor fusion architecture.