Explainability Evaluation of Transfer Learning Models Utilized in Multimodal Translation of Sign Language

Sign language is a mode of communication for people with hearing loss and speaking disabilities; however, gesture meanings vary from region to region. The proposed study evaluates the efficacy of transfer learning for converting sign gestures into natural language text and audio. American Sign Language and Sinhala Sign Language datasets were used to demonstrate the impact of data augmentation and image preprocessing on model performance across RGB, grayscale, and binary formats. Transfer learning models, including MobileNet, InceptionV3, DenseNet121, and DenseNet201 were utilized, and results indicate that RGB data significantly outperforms other formats. MobileNet achieved the highest accuracies of 98.62% for the ASL dataset and 99.66% for the Sinhala Sign dataset. Real-time validation was conducted via webcam inference to ensure practical reliability. Furthermore, SHAP (SHapley Additive exPlanations) was used to evaluate the proposed model's interpretability. The study concludes that combining lightweight transfer learning architectures with robust data augmentation can lead to significant improvements in communication of sign language users.