Vision-based Sign Language Translation via a Skeleton-Aware Neural Network

Sign languages are mainly expressed by human actions, such as arm, hand and finger motions. Thus skeleton which reflects human pose information can provide an important cue to distinguish signs (i.e., human actions), and can be used for Sign Language Translation (SLT), which aims to translate sign language to spoken language. However, the recent neural networks typically focus on extracting local-area or full-frame features, while ignoring informative skeleton features. Therefore, this paper proposes a novel Skeleton-Aware neural Network (SANet) for vision-based SLT. Specifically, to introduce skeleton modality, we design a self-contained branch for skeleton extraction. To efficiently guide the feature extraction from video with skeletons, we concatenate the skeleton channel and RGB channels of each frame for feature extraction. To distinguish the importance of clips (i.e., segmented short video), we construct a skeleton-based Graph Convolutional Network (GCN) for feature scaling, i.e., giving importance weight for each clip. Finally, to generate spoken language from features, we provide an end-to-end method and a two-stage method for SLT, respectively. Besides, based on SANet, we also provide a SLT solution on the smartphone for benefiting communication between hearing-impaired people and normal people. Extensive experiments on three public datasets and case studies in real scenarios demonstrate the effectiveness of our approach, which outperforms the existing methods.