A Pipeline of Energy Efficient Action Detection

Abstract

Action detection has been an essential topic in computer vision tasks for the last decade. There is lots of research done to get high accuracy in action detection based on image features. However, image features consume heavy computation and energy. For those models deployed on edge devices, image features are not very suitable and applicable. A less computational method based on the skeleton is proposed by [1]. Compared to image features, skeleton features require significantly fewer flops. Nonetheless, most skeleton-based action detection research focuses on existing datasets. Little research has been put into running action detection on mobile devices. In this thesis, a pipeline in energy-efficient action detection is proposed to achieve action detection on unmanned aerial vehicles (UAVs). The environmental platform is Raspberry Pi. This pipeline could detect person real-time on Raspberry Pi and classify action through a skeleton-based action classification network. There is lots of research in this area for object detection, including deploying models on edge devices. However, most research considers multi-classes and generalize, and the network structure is not light enough in this situation. This thesis proposes a specific detection network with higher FPS but lower flops. Besides, a skeleton-based spatial-temporal transformer network is also proposed in this thesis. Action classification network consists of a graph convolution network and a multi-head transformer module. Gaussian distribution is introduced as weak supervision in action classification. The pipeline in this thesis consists of three parts: proposed object detection network, existing light pose estimation network, and skeleton-based action classification network. This pipeline achieves an end-to-end structure on mobile devices. To speed up inference, this thesis also prunes and quantizes each model. This pipeline has been tested with deployment on a Raspberry Pi and videos recorded by UAV under a public environment. An energy-measuring system is established to measure energy consumption.