MACHINE LEARNING MODEL FOR RECOGNITION OF UKRAINIAN SIGN LANGUAGE HAND CONFIGURATIONS

Abstract

Background. Sign language recognition using Artificial Intelligence (AI) is a relevant and still unsolved problem. For Ukrainian Sign Language (USL), the task is complicated by a lack of publicly available datasets. Consequently, researchers operating under data-constrained conditions should apply additional features for improving model accuracy. Hand configuration is a fundamental characteristic of lexical units in any sign language. In USL 40 configurations exist. The ability to accurately identify configuration would provide relevant features for AI-based sign language recognition systems.

Objective. To develop a machine learning model capable of frame-by-frame recognition of USL hand configurations on a video stream with an accuracy of at least 95% on the test set and an average inference time not exceeding 70 ms.

Methods. As input data for the USL hand configurations recognition models, a system of mathematical indicators, calculated from the keypoints coordinates, extracted via MediaPipe pose estimation system, was proposed. The training dataset was created from camera images of USL hand configurations performed by middle-aged subjects. Following keypoints extraction through MediaPipe, the corresponding vectors of mathematical indicators were computed. Various architectures of random forest and perceptron were evaluated.

Results. The highest performance was achieved by the perceptron with two hidden layers of 300 neurons each, which takes the vector of proposed mathematical indicators as input. The model reached an accuracy of 98.18% on the test set. During testing on a video stream, it identified without error 32 out of 40 USL configurations, and with only sporadic deviations. The average inference latency was 46 ms. Notably, a perceptron of identical architecture trained on raw MediaPipe keypoint coordinates exhibited lower accuracy on a video stream. Furthermore, the proposed indicators improved random forest model performance, confirming their universality.

Conclusion. The optimal model among the considered architectures was a perceptron with two hidden layers of 300 neurons each, which takes the vector of proposed mathematical indicators as input. It identifies in real-time USL hand configurations, achieving an accuracy of 98.18% on the test set. In experiments based on the proposed expert assessment criterion on the video stream, it correctly identified 35% more configurations (80% vs. 45%), compared to the model based on raw keypoint coordinates. Therefore, implementing additional verification based on expert assessment in addition to the standard accuracy metric on the test data is effective. The presented architecture is the first artificial intelligence-based model for the USL hand configuration recognition. Future research will focus on integrating this network into a system for isolated recognition of USL.