Towards Efficient Deep Learning: From Compression, Search to Unification

Abstract

Deep learning has gained considerable interest due to its record-breaking performance in a variety of different domains, including computer vision, natural language processing, multimodal understanding, etc. Meanwhile, deep neural networks are usually parameter-heavy, inefficient, and highly specialized. As a result, there has been a growing demand to improve the efficiency and interoperability of deep neural networks motivated by different needs. In this dissertation, we proposed to address those problems via serial of approaches, including (a) reducing the memory storage and energy footprint via parameter sharing (b) improving the trade-off between performance and computation via neural architecture search (c) unifying neural architectures across different modalities via cross-modality gradient harmonization.