Mobile QR Code QR CODE : The Transactions of the Korean Institute of Electrical Engineers
Title Hand Gesture Classification Using Early Fusion Based Multimodal Deep Learning
Authors 김익진(Ik-Jin Kim) ; 김수열(Su-Yeol Kim) ; 이용찬(Yong-Chan Lee) ; 이연정(Yun-Jung Lee)
DOI https://doi.org/10.5370/KIEE.2021.70.11.1714
Page pp.1714-1721
ISSN 1975-8359
Keywords Hand Gesture Classification; Deep Learning; EMG; Multimodal Learning; Ninapro DB
Abstract In this paper, we propose a new hand gesture classification strategy using early fusion based multimodal deep learning. The structure and parameters of the state-of-the-art deep learning models such as ResNet152, DenseNet201, EfficientNetB0 for the source task of image classification are reused in the target task of hand gesture classification using surface electromyograph(EMG) and finger's kinematic data. The time-domain EMG and kinematic signals are normalized and then transformed into combined 2-D images for the early-fusion network. The experimental results support the superiority of the proposed method in terms of classification accuracy. The transfer learning model with the EfficientNetB0 shows the 93.94% accuracy for 40 gestures of 40 participants in the Ninapro DB2.